This is a fun project I did for my computer vision research project at the University of Canterbury . It uses the Kinect face-tracking library to replace a user’s face with a custom face, which warps to match their expression.

It is written in C++ and includes a few GLSL shaders to blend the model face with the natural scene lighting.

Download

A stand-alone version is available here if you want to try it out:

face-replace.zip (12MB)

You’ll need to download the Microsoft Kinect SDK Runtime which installs the required device drivers for the Kinect. Alternatively you can use a different depth sensor, as long as it works with OpenNI2.

Compiling

I developed this project in Visual Studio 2013 (MSVC++ 12), but it should work in other versions provided you can compile the required libraries.

Before you can compile this, make sure you set up the following libraries:

Operation

The program performs the following steps:

  1. Capture RGB + Depth video frame
  2. Detect head pose and face features using Kinect SDK
  3. Deform the Candide-3 mesh to the given head pose and face features
  4. Process the RGB + Depth frames using OpenCV
  5. Draw the RGB video frame
  6. Draw the texture-mapped candide-3 model in OpenGL, using a custom blend shader.

Side-note: This project uses a custom candide-3 face model instead of the Kinect SDK’s internal model, since it’s not easy to match vertices with tex coords using the internal model. This functionality is provided through the WinCandide-3 project (all source code named ’eru’ is part of this project).

Future Work

It’s probably unlikely I’ll do much more on this project since I have other commitments, but here’s a list of things that could be improved upon in the future:

  • Write a plugin for blender that can read and write the candide-3 model, so textures can be more accurately mapped. (I’m currently using the WinCandide-3 utility to approximately map the texture)
  • Add support for multiple people
  • Decrease tracking latency/improve face location. (Perhaps something like meanshift/optical flow + a kalman filter?)

Source Code

Source code is available on my GitHub page: face-replace

My professional paper is available here: COSC428 Computer Vision - FaceReplace.pdf

comments powered by Disqus