Unity3d + OpenCV – Face tracking – Head POV

Torino (Italia)
Many unity developers tried to integrate the OpenCv library inside Unity… I wanted too, so I done it.

Thanks to Aimino that gives me the pre-compiled OpenCVSharp dll.

The worst thing of the integration was that the Mono version used in Unity doesn’t support the System.Drawing namespace that are widely used in the c# wrapper of OpenCV. So the porting was be very painful but thanks again to Aimino I could concentrate myself in creating a handle class to take care of the IplImage to Texture conversion and viceversa and find a fast solution to achieve acceptable performance.

Many of the test that i saw on youtube or in the Unity forum were using some workaround like creating the OpenCv application that sent through messaging protocol (like OSC, TSPS, ecc…) the opencv information to Unity. Other approaches were integrating the .dll directly inside Unity as plugin. The first result was discouraging… 4-5 fps for a simple conversion of a video stream to IplImages to texture. Working around the possible optimization I reached an acceptable frame rate of 90 fps, with an OpenCv capture stream.
I also test the viceversa conversion using the Unity webcamtexture as input, converted in the IplImages format and than used to compute interesting OpenCv stuff, the results was again around 100 fps.

Enjoy it!.

Share this:
Facebook Twitter Plusone Email

19 Comments

  • Great job! Could you plz send me the dll file, i’m looking for this for a long long time.
    Thx

  • Hello, I am working on a project and I am trying to integrate Unity and OpenCV but I am stuck trying to convert between Texture and IplImage, could you give me some hints on this?

    • Hello Matias,
      The IplImage, coming from a capture device, are encoded in a BGR format, the Unity Texture2d in RGBA, so you need to make a for cycle to assign the correct corresponding channels and set to 0 (or 1 it depends on your purpose) the alpha for every pixels in the image.
      Hope it will help.

      • I still cannot make it work, could you share this particulare piece of code? (converting from IplImage to Texture2D)

        • I also want to know the detail.

      • I managed to convert IplImage into Texture2D by assigning the RGB value of each pixel,
        but the performance is not really good,
        I only achieve around 10~20 fps,
        any tips on improving the performance?

        Many thanks.

  • Hi,
    I wanto to experiment with unity and opencv also…can you send me the pre-compiled OpenCVSharp dll?

  • Heey!
    Great work you’ve done! Do you mind to share how you have done the converting part from IplImage to texture2d?
    regards!

  • Hi,Sir, I tried set the Unity WebcamTexture to IglImage format,But Just Only 3~4 Fps , So Can you provide line or way.Thanks.
    ps: I’m apply OpenCVsharp2.4.0 and webcam640*480.

  • Hi dude,

    I have used the same starting off project as you but noticed when i tried to turn the CaptureTexture scene into a build for pc it doesnt work.

    I tracked down the error to a dll file that isnt copied over when you create the build.

    Have you tried to do a build with unity3d

  • Hi,

    I am looking at the same problem.I want to use OpenCV within Unity3d and use it with Android and IOS devices.I am not sure if this has a good performance or not… but I can’t get it running since, as you already mentioned, System.Drawing namespace is not provided by the Mono fork unity is using.
    I am wondering if you can give me some hints how you exactly managed to avoid using this namespace.
    Any help is highly appreciated.

  • Hi Ivano, I started to work with OPENCVsharp pre-compiled in Unity…However, i cant access to some functions like BitmapConverter. Can you help me? I’m converting iplImage to Texture2D

    I have dll files in unity project as follows:
    - In Assets folder: OpenCvSharp.dll and OpenCvSharp.MachineLearning.dll
    - In root folder: cv210.dll, cvaux210.dll, cxcore210.dll, highgui210.dll, ml210.dll, OpenCvSharpExtern.dll

  • Hi Matias,

    I’m trying to convert IplImage to Texture2D and viceversa inside Unity. I really appreciate if you can share the source code for that.

    Cheers.

  • Hi Ivan

    will you please explain to me how you when about to incorporate opencv and unity? I have done my application (face recognition and eyeglasses fit) in opencv and VS now for a nice looking GUI i want to integrate it to unity. What are the step required? please i really need your help.
    Thanks
    Parfait

  • Hi,

    Could you send me .dll file please?
    I would like to try

    Thank you,
    juanfry

  • Hey, can you give me OpenCVSharp.dll
    Please sent me on my e-mail.
    Thank you

  • Hi,

    I am just starting on unity3d and trying to integrate opencv.
    I am facing problem in converting the captured image into texture, can you help me out by sharing the piece of code with me.

    Thank You
    Ayan

  • Hello friend. Great job! Is the dll avaliable for us to try it out?

  • Hello. I also do the head-tracking perspective stuff with Unity. I use Kinect for the head tracking.
    But it is not convenient some time. Could I purchase your head-tracking with webcam package in asset store?

    Tom Tong
    Creattive Coder From Hong Kong

Leave a comment