Christina Gsaxner

Augmented Reality, Computer Vision, Medical Imaging

3 minute read

You can run computer vision algorithms written in C++ supported by the newest OpenCV library directly on the HoloLens 2. This is useful if you, e.g., want to process frames from one of the HoloLens’ on-board sensors directly on the device. One way to do this is via a C++/WinRT Windows Runtime Component, which can be consumed by a Unity App for the HoloLens.

This tutorial will cover how to create such a Plugin.

Requirements

  • Your desired Version of OpenCV, built for ARM64/UWP. Check out my Tutorial on how to build OpenCV yourself!

  • Visual Studio (Community version is fine).

    • Make sure you have the following workloads:
      • Desktop development with C++
      • Universal Windows Platform development -> including a Windows 10 SDK
      • .NET desktop development
      • Game development with Unity (if you want to use Unity)
    • Also, the following individual components should be included:
      • USB Device Connectivity
      • C++/WinRT (pre-installed when using VS2022)
      • Visual C++ compilers and libraries for ARM64/x86
      • C++ Universal Windows Platform tools for ARM64

Configuring the Plugin in Visual Studio

  • Open Visual Studio and create a new Project. Use the Windows Runtime Component (C++/WinRT) for UWP template.

  • Since the HoloLens has a Windows 10 based operating system, we need to target Windows 10 as a platform.

  • Make sure your configuration is set to “Release” and platform to “ARM64”.

  • Right-click on your project in the Solution Explorer, and open the Properties page.

  • Navigate to C++ -> General tab and open the Additional Include Directories entry. Add the path to your OpenCV include directory at <opencv_build>/install/include.

  • Navigate to Linker -> General tab and open the Additional Library Dependencies entry. Add the path to your OpenCV lib directory at <opencv_build>/install/include/ARM64/vc<compiler_version>/lib.

  • Navigate to Linker -> Input tab and open the Additional Dependencies entry. Add all OpenCV module libraries you want to include in your project, e.g. opencv_core<opencv-version>.lib.

  • Click on “Apply” and “OK” for the changes to take effekt.

  • Finally, open the pch.h file in your Solution. Include the header files to the OpenCV modules you want to use here, e.g., #include <opencv2/opencv.hpp>.

  • Build your project to make sure everything has been configured correctly.
  • Now, you can implement your algorithms using OpenCV!

Plugin and Unity Interop

In the Plugin solution

  • Within the Plugin solution, you use the .idl file to define the interface between Unity and the Plugin. I.e., you will be able to call the functions and access the properties defined in your .idl file from a C# script in Unity. You can also declare constructors in the .idl, which means you will be able to activate them from the outside. E.g.: 
    // MyRuntimeClass.idl
namespace MyProject
{
    runtimeclass MyRuntimeClass
    {
        MyRuntimeClass();
        Int32 MyProperty;
    }
}
  • The implementation happens in the corresponding .h and .cpp files: 
    // MyRuntimeClass.h
namespace winrt::MyProject::implementation
{
    struct MyRuntimeClass : MyRuntimeClassT<MyRuntimeClass>
    {
        MyRuntimeClass() = default;

        int32_t MyProperty();
        void MyProperty(int32_t value);
    };
}
  • After you build your Plugin solution, you will find your Plugin files MyProject.dll and MyProject.winmd files under MyProject/ARM64/Release/MyProject/.

In Unity

  • In your Unity project, create a new folder Assets/Plugins/WSA/ARM.
  • Drag-and-drop your MyProject.dll and MyProject.winmd to this folder.
  • Furthermore, drag-and-drop any OpenCV .dlls your project depends on to this folder, e.g., opencv_core480.dll.
  • Now, you can consume the Plugin in any C# script, if the project has Windows Runtime support enabled: 
    using UnityEngine;
#if ENABLE_WINMD_SUPPORT
using MyProject;
#endif

public class MyUnityClass : MonoBehaviour 
{
  void Start()
#if ENABLE_WINMD_SUPPORT
      MyRuntimeClass runtimeClass = new MyRuntimeClass();
      int myProperty = runtimeClass.MyProperty;
#endif
}

That’s it! Now you can run every OpenCV-based algorithm directly on the HoloLens. Using the HoloLens2 Research Mode API, you can process frames from the HoloLens RGB camera, depth sensor and tracking cameras directly within the Plugin.

You May Also Enjoy