Monday, January 27, 2014

Changing the way .net looks for dependencies

Whenever I have to clean up my machine in preparation for changing my hardware or an OS refresh, I always find some forgotten gems in the process. Here’s one that I’m throwing up onto my blog for prosperity sake.

Every so often I have a project with lots of assemblies and dependencies which results in a deployment folder that can be daunting to the uninitiated. It would be great if the deployment folder had the exe and the config file and all the other dependencies were hidden in subfolders.

Here’s an example of how to do that.

  1. First, create two projects. Shell is an exe and Dependency is a ClassLibrary
  2. Change the Build folder of Shell to point to “..\AssemblyOutput”
  3. Change the Build folder of Dependency to point to “..\AssemblyOutput\modules”
  4. In the Shell project, add a reference to the Dependency project
  5. In the Shell’s references, set the CopyLocal property of the Dependency reference to “False”. If you don’t do this, the assembly will be copied to the root folder of the Shell’s output folder.
  6. Add the following to the Shell’s app config:

    <assemblyBinding xmlns="urn:schemas-microsoft-com:asm.v1">
      <!-- Probing path is relative to the executable. Multiple values are supported (bin;bin2;modules;modules\subfolder)-->
      <probing privatePath="modules"/>



Now when you run the project, the first time the code comes across a reference to a type inside the Dependency assembly, it will load the assembly in the modules folder.

Happy coding.

submit to reddit

Tuesday, January 14, 2014

Leverage the Kinect v2 underlying depth buffer

One of the great new features in the API for new Kinect for Windows v2 developer preview is the ability to access the underlying buffer for the data sources. Normally, you can get by with using the CopyFrameDataToArray method but this option allows you to get a little closer to the metal and possibly squeeze out a bit more performance. Frankly, this is a good thing since we’re iterating over 217,088 pixels at a rate of 30 times per second – anything helps.

This is an early preview of the new Kinect for Windows, so the device, software and documentation are all preliminary and subject to change.

Also note, because C# does not support pointer arithmetic by default, you must mark your methods with the unsafe keyword.

To access the buffer, we use the AccessUnderlyingBuffer method on the DepthFrame which tells us how big the size of the buffer is and a pointer to the buffer.

    uint size = 0;
    IntPtr ptr = IntPtr.Zero;

    depthFrame.AccessUnderlyingBuffer(out size, out ptr);

    // ptr now points to the buffer

Once we have a pointer to the buffer we can iterate through it’s contents. It’s up to us to understand how to parse the buffer and it’s not type-safe. We assume that the buffer is comprised of an array of ushort.

// this is also unsafe, so keep in the unsafe block above...
// obtain a pointer to the ushort array
ushort* data = (ushort*)ptr.ToPointer();

for(int i = 0; i < size; i++)
    ushort depth = data[i];

    // work with depth data

In my modest experimentation, switching to the buffer reduced cpu time between 0.5 – 1.0 milliseconds. Your mileage will obviously vary. Here’s a small clip of me toggling between the two approaches. The CPU counter in the bottom left is averaged over the number of frames per second and includes the time to render and update the image. There’s a subtle difference.

Happy Coding!

submit to reddit

Friday, January 10, 2014

Finding Pixels for Bodies in Kinect for Windows v2

In this post, I’m using Depth and BodyIndex data to create a depth visualization that highlights the active users.

This is an early preview of the new Kinect for Windows, so the device, software and documentation are all preliminary and subject to change.


This post illustrates a few breaking changes from v1 of the Kinect for Windows API.

First off, the event signature to receive all data from the sensor has changed. Rather than getting data from the AllFramesReady event of the sensor, version 2 now uses a MultiSourceFrameReader that has a MultiSourceFrameArrived event. The MultiSourceFrameReader allows you to specify which data streams you wish to receive. This is a relatively minor change.

The other minor change this post illustrates is that the depth data no longer contains a byte to indicate the player index. Rather than applying a bitmask to the depth data, it is now exposed as its own data stream.

this.sensor = KinectSensor.Default;
this.reader = sensor.OpenMultiFrameSourceReader(
                        FrameSourceTypes.BodyIndex |
this.reader.MultiSourceFrameArrived += MultiSourceFrameArrived;

DepthFrameDescription desc = sensor.DepthFrameSource.FrameDescription;

this.bytesPerPixel = (PixelFormats.Bgr32.BitsPerPixel + 7) / 8;
this.depthFrameData = new ushort[desc.Width * desc.Height];
this.bodyIndexData = new byte[desc.Width * desc.Height];

this.pixels = new byte[desc.Width * desc.Height * this.bytesPerPixel];
this.bitmap = new WriteableBitMap(desc.Width, desc.Height, 96, 96, PixelFormat.Bgr32, null);

When the frame arrives, I fill both the depth and body index arrays with the available data. All that's left is to loop through the depth data and color the pixels. In this case, I'm using the GradientColorMap from my last post to decorate color pixels differently than background pixels.

void MultiSourceFrameArrived(object sender, MultiSourceFrameArrivedEventArgs e)
    MultiSourceFrameReference frameReference = e.FrameReference;

    MutliSourceFrame multiSourceFrame = null;
    DepthFrame depthFrame = null;
    BodyIndexFrame bodyIndexFrame = null;

        multiSourceFrame = frameReference.AcquireFrame();
        if (multiSourceFrame != null)
            var depthFrameReference = multiSourceFrame.DepthFrameReference;
            var bodyIndexFrameReference = multiSourceFrame.BodyIndexFrameReference;

            depthFrame = depthFrameReference.AcquireFrame();
            bodyIndexFrame = bodyIndexFrameReference.AcquireFrame();

            if ((depthFrame != null) && (bodyIndexFrame != null))

                int pixelIndex = 0;
                for (int i = 0; i < depthFrameData.Length; ++i)
                    ushort depth = depthFrameData[i];
                    byte player = this.bodyIndexFrameData[i];
                    Color pixelColor;

                    if (player == 0xff) // nobody here
                        pixelColor = this.grayscaleMap.GetColorForDepth(depth);
                        pixelcolor = this.playerMap.GetColorForDepth(depth);
                pixels[pixelIndex++] = pixelColor.B;
                pixels[pixelIndex++] = pixelColor.G;
                pixels[pixelIndex++] = pixelColor.R;

                 new Int32Rect(0, 0, depthWidth, depthHeight),
                 depthWidth * this.cbytesPerPixel,


        // MultiSourceFrame, DepthFrame, ColorFrame, BodyIndexFrame are IDispoable
        if (depthFrame != null)
           depthFrame = null;

        if (bodyIndexFrame != null)
            bodyIndexFrame = null;

        if (multiSourceFrame != null)
            multiSourceFrame = null;


Easy peasy! Happy Coding!

Wednesday, January 08, 2014

Accessing Depth data with Kinect v2

This is an early preview of the new Kinect for Windows, so the device, software and documentation are all preliminary and subject to change.

Accessing Depth Data for the new Kinect for Windows API v2 is easy and very similar to the previous version of the API. We’ll do some more with this in a bit, but here’s a code snippet that’s been paraphrased from the v2 samples:

// Get the Sensor
this.sensor = KinectSensor.Default;

if (this.sensor! = null)
    // If we have a sensor, open it

    // this example will use a BGR format image...    
    this.bitsPerPixel = (PixelFormats.Bgr32.BitsPerPixel + 7) / 8;

    // initialize some arrays based on the depth frame size
    FrameDescription desc = this.sensor.DepthFrameSource.FrameDescription;
    int size = desc.Width * desc.Height;
    this.frameData = new ushort[size];
    this.pixels = new byte[size * bitsPerPixel];

    this.bitmap = new WriteableBitmap(desc.Width, desc.Height, 96.0, 96.0, PixelFormats.Bgra32, null);
    // obtain a reader and subscribe to events
    this.reader = this.sensor.DepthFrameSource.OpenReader();
    this.reader.FrameArrived += FrameArrived;

void FrameArrived(object sender, DepthFrameArrivedEventArgs e)
    DepthFrameReference frameReference = e.FrameReference;

        DepthFrame frame = frameReference.AcquireFrame();

        if (frame != null)
            // frame is disposable

                var desc = frame.FrameDescription;


                int colorPixelIndex = 0;

                // loop through the frame data
                for(int i = 0; this.frameData.Length; ++i)
                    ushort depth = this.frameData[i];

                    // do something interesting with depth data here
                    // see examples below...
                    // Color color = this.colorMap.GetColorForDepth(depth);
                    Color color = new Color();

                    // assign values to individual pixels
                    // BGRA format
                    this.pixels[colorPixelIndex++] = color.B; 
                    this.pixels[colorPixelIndex++] = color.G;
                    this.pixels[colorPixelIndex++] = color.R;
                    ++colorPixelIndex; // skip Alpha color

                    new Int32Rect(0, 0, desc.Width, desc.Height),
                    desc.Width * this.bitsPerPixel,

        // frame might no longer be available

Fairly straight-forward, right? The biggest difference from v1 is that we must access the data from an event on the reader rather than an event from the sensor.


The example in the SDK truncates the depth data down to the last byte, which produces a grayscale effect that “wraps” every 255 millimeters. It doesn’t look very nice, but the key thing to observe is just how detailed the depth data is. If you look closely, you can clearly make out facial features and other fine details such as the lines in my hands.


// paraphrased from the SDK examples..
// truncates the depth data down to the lowest byte, which
// produces an "intensity" value between 0-255

ushort minDepth = desc.DepthMinReliableDistance;
ushort maxDepth = desc.DepthMaxReliableDistance;

// To convert to a byte, we're discarding the most-significant
// rather than the least-significant bits.
// We're preserving detail, although the intensity will "wrap."
// Values outside the reliable depth range are mapped to 0 (black).
byte intensity = (byte)(depth >= minDepth && depth <= maxDepth) ? depth : 0;

// convert intensity to a color (to fit into our example above)
var color = new Color()
            R = intensity,
            G = intensity,
            B = intensity

Prettier examples

Here’s a few examples I’ve cooked up. From past experience with the Kinect v1 we observed that you can improve performance by reducing the amount of calculations per frame. One approach I’ve used it to pre-calculate the results into a hash table between 500 and 4500 milimeters. There are likely other optimizations that can be made, but this is easy.

For fun, I’ve created an abstraction called a DepthMap:

abstract class DepthMap
    protected ushort minDepth;
    protected ushort maxDepth;

    protected DepthMap(ushort minDepth, ushort maxDepth)
        this.minDepth = minDepth;
        this.maxDepth = maxDepth;

    public abstract Color GetColorForDepth(ushort depth);

HSV Color Wheel

Rather than a grayscale effect, I found a few great online samples on how to implement a HSV color wheel which produces this fun rainbow effect. I’m gradually transitioning between colors by looking at the most significant bits.


double h, s, v;
int r, g, b;

h = ((depth - minDepth) / (maxDepth - minDepth)) * 360;
v = 1;
s = 1;

Hsv.HsvToRg(h, v, s, out r, out g, out b);

return new Color {
        R = (byte)r,
        G = (byte)g,
        B = (byte)b

Color Gradients

The formula to produce a gradient is rather simple, you take a percentage of one color and a remaining percentage of another color. This color map allows me to pick the colors I want to use, which produces some interesting effects. Using an array of colors (Red, Yellow, Green, Yellow, Red) I can color code depth to reflect an ideal position from the camera.


var colorMap = new GradientColorMap(
    new[] { 

I can also create a boring grayscale effect by just using Black and White.

var colorMap = new GradientColorMap(
    new[] {

You can see some of them in action here:

Happy coding!

Monday, January 06, 2014

Multiple Apps can use Kinect for Windows v2

One of the new features of the Kinect for Windows API is that multiple applications can simultaneously access the sensor, as demonstrated by this video:

This is an early preview of the new Kinect for Windows, so the device, software and documentation are all preliminary and subject to change.

The lower right hand corner of this video shows a console application, KinectService, that must be running in order to run the samples provided. It’s unclear what role the service will play in the future releases of the SDK but I’d expect that it runs as Windows Service or headless process running from the system tray.

The Kinect API has changed to reflect this ability. It’s a minor change that will break some of your existing v1 code, but should be relatively easy to port to v2. Previous versions of the API allowed you to access the sensor data by subscribing to events on the sensor. The new version of the API introduces a reader pattern that separates the events from the sensor. Each reader has access to their own frames, which should in theory allow for more modular applications.

Here’s a quick sample:

// v1
KinectSensor sensor = KinectSensor.Sensors[0];
sensor.DepthFrameReady += OnDepthFrameReady;

// v2
KinectSensor sensor = KinectSensor.Default;
DepthFrameReader reader = sensor.DepthFrameSource.OpenReader();
reader.FrameArrived += OnDepthFrameArrived;

This change is significant because most devices are exclusively locked to the application that is using it. This can be a real pain when your app crashes but doesn’t release the handle of the device.

I can see many options for this functionality:

  • A support engineer can remotely troubleshoot the environment setup without having to close the primary application
  • A background process could take periodic snapshots to produce timelapse videos of usage of your application
  • Applications produced by different vendors could simulatenously run on the same machine: A desktop application could dim the monitor when you're not looking at it while a gesture application augments your windows experience...

What will you do with this feature?