Tuesday, September 30, 2014

Mobile Services: Apply Database Migrations on Startup

As per my last post, if you’re using Entity Framework with Azure Mobile Services you’ll want to move away from Automatic Database upgrades and move to Code Based migrations. There are several strategies posted online, here’s the strategy that worked best for our project.

One of the challenges that my team identified early in the process was that our database content changed more frequently than our database schema. The default database initializer strategies only run the database migration and seeder scripts when the schema changes, so we found it more useful to manually apply the database migrations on start-up. This adds a bit of performance overhead to the start-up process, but since our database seeder scripts were relatively small, this seemed like an acceptable trade-off.

First, you’ll want to enable Code Based Migrations if you haven’t already. This involves opening the Package Manager Console window and typing the following:

Enable-Migrations

This power-shell command will add a few additional NuGet packages and make a few changes to your project. Most notably, the script will add a Migrations folder to your project and a Configuration class. We’ll make a few changes to the Configuration class by allowing automatic migrations and allowing data loss. The data loss feature will allow you to drop database columns even if they have data, so pay attention to your migration scripts.

internal sealed class Configuration : DbMigrationsConfiguration<MyServiceContext>
{
  public Configuration()
  {
    ContextKey = "MyProject.Models.MyServiceContext";
   
    AutomaticMigrationsEnabled = true;
    AutomaticMigrationDataLossAllowed = true;
  }

}

Next you’ll want to override the Seed method and populate your database with starter data. I highly recommend moving all your seeding logic to a static method so that you can reuse this logic in your test scripts, etc. Also note that the System.Data.Entity.Migrations namespace adds an AddOrUpdate extension method for DbSet<T> which greatly simplifies your database seed. Following this approach will allow you to run the seed as many times as you want without introducing duplicates.

protected override void Seed(MyServiceContext context)
{
    //  This method will always be called.

    //  You can use the DbSet<T>.AddOrUpdate() helper extension method 
    //  to avoid creating duplicate seed data. E.g.
    //
    //    context.People.AddOrUpdate(
    //      p => p.FullName,
    //      new Person { FullName = "Nigel Tufnel" },
    //      new Person { FullName = "David St. Hubbins" },
    //      new Person { FullName = "Derek Smalls" }
    //    );
    //
    MyDatabaseSeeder.Seed(context);
}

Finally, to apply the database migrations on start-up you’ll need to add the following to your WebApiConfig.Register() method:

public static void Register()
{
   
   Database.SetInitializer<MyServiceContext>(null);

   var migrator = new DbMigrator(new Configuration());
   migrator.Update();

}

With this in place, the DbMigrator will apply any database changes when there are available and reliably call the Seed method with every deployment.

Happy Coding.

submit to reddit

Monday, September 15, 2014

Mobile Services: Sharing a database schema between environments

I recently completed a project where we used Azure Mobile Services as the backend system for an Android application. I was pleased at how easy it was to setup and deploy to an environment, but there are a few details for production deployments that make things a bit tricky.

As your development team sprints full steam towards the finish line one major hurdle you’ll have to cross is database versioning. The default entity framework strategy is to use a database initializer that drops your database on start-up or any time the model changes, which obviously is not good and you can’t deploy to production with this enabled. The solution for this is to disable automatic upgrades and use code-first database migrations, which are also a bit tricky (I might blog about later). Here’s an initial primer on database migrations.

Before you run off to enable database migrations and start scripting out your objects into code, there’s something you should know. If you’re like me and you have separate azure environments for testing and production, the scripted objects will contain the schema name of your development environment and you’ll most likely get an error about the “__MigrationHistory” table on start-up.

You can avoid this hassle by making a small change to your DatabaseContext to ensure both environments use the same schema. The default boilerplate code uses the name of the Azure Mobile Service in your application-settings:

// part of MyMobileServiceContext : DbContext
protected override void OnModelCreating(DbModelBuilder modelBuilder)
{
    string schema = ServiceSettingsDictionary.GetSchemaName();
    if (!string.IsNullOrEmpty(schema))
    {
        modelBuilder.HasDefaultSchema(schema);
    }

    modelBuilder.Conventions.Add(
        new AttributeToColumnAnnotationConvention<TableColumnAttribute, string>(
            "ServiceTableColumn", (property, attributes) => attributes.Single().ColumnType.ToString()));
}

This simple change to use the same schema name ensures that your scripted objects and runtime time stay in sync:

protected override void OnModelCreating(DbModelBuilder modelBuilder)
{
    //string schema = ServiceSettingsDictionary.GetSchemaName();
    string schema = "MySchemaName";
    if (!string.IsNullOrEmpty(schema))
    {
        modelBuilder.HasDefaultSchema(schema);
    }

    modelBuilder.Conventions.Add(
        new AttributeToColumnAnnotationConvention<TableColumnAttribute, string>(
            "ServiceTableColumn", (property, attributes) => attributes.Single().ColumnType.ToString()));
}

Happy coding.

Wednesday, March 26, 2014

Head Tracking with Kinect v2

This is yet another post in my series about the new Kinect using the November 2013 developer preview SDK. Today we’re going to have some fun by combining the color, depth and body data streams (mentioned in my last few posts, here, here and here) and some interesting math to create an image that magically tracks the user’s head.
This is an early preview of the new Kinect for Windows, so the device, software and documentation are all preliminary and subject to change.
ColorExample-09-29-58
If you recall from my last post, I used the CoordinateMapper to translate the coordinates of the user’s joint information on top of the HD color image. The magic ingredient converts the Joint’s Position to a ColorSpacePoint.

Joint headJoint = body.Joints[JointType.Head];

ColorSpacePoint colorSpacePoint = 
    _sensor.CoordinateMapper.MapCameraSpaceToColorPoint(headJoint.Position);
If we take the X & Y coordinates from this ColorSpacePoint and the wonderful extension methods of the WriteableBitmapEx project, we can quickly create a cropped image of that joint.
int x = (int)Math.Floor(colorSpacePoint.X + 0.5);
int y = (int)Math.Floor(colorSpacePoint.Y + 0.5);

int size = 200;

WriteableBitmap faceImage = _bmp.Crop(new Rect(x,y,size,size));
ColorExample-08-04-32
Wow, that was easy! Although this produces an image that accurately tracks my head, the approach is somewhat flawed as it doesn’t scale based on the user’s position from the camera: if you stand too close to the camera you’ll only see a portion of your face; Stand too far and you’ll see my face and torso. We can fix this by calculating the desired size of the image based on the depth of the joint. To do this, we’ll need to obtain a DepthSpacePoint for the Joint and a simple trigonometric formula…
The DepthSpacePoint by itself doesn’t contain the depth data. Instead, it contains the X & Y coordinates from the depth image which we can use to calculate the index in the array of depth data. I’ve outlined this in a previous post, but for convenience sake here’s that formula again:
// get the depth image coordinates for the head
DepthSpacePoint depthPoint =
    _sensor.CoordinateMapper.MapCameraPointToDepthSpace(headJoint.Position);

// use the x & y coordinates to locate the depth data
FrameDescription depthDesc = _sensor.DepthFrameSource.FrameDescription;
int depthX = (int)Math.Floor(depthPoint.X + 0.5);
int depthY = (int)Math.Floor(depthPoint.Y + 0.5);
int depthIndex = (depthY * depthDesc.Width) + depthX;

ushort depth = _depthData[depthIndex];
To calculate the desired size of the image, we need to determine the width of the joint's pixel in millimeters. We do this using a blast from the past, our best friend from high-school trigonometry, Soh-Cah-Toa.
Kinect_Depth
Given that the Kinect’s Horizontal Field of View is 70.6°, we bisect this in half to form a right-angle triangle. We then take the depth value as the length of the adjacent side in millimeters. Our goal is to calculate the opposite side in millimeters, which we can accomplish using the TOA portion of the mnemonic:
tan(0) = opposite / adjacent
opposite = tan(0) * adjacent
Once we have the length of the opposite, we divide it by the number of pixels in the frame which gives us the length in millimeters for each pixel. The algorithm for calculating pixel width is shown here:
private double CalculatePixelWidth(FrameDescription description, ushort depth)
{
    // measure the size of the pixel
    float hFov = description.HorizontalFieldOfView / 2;
    float numPixels = description.Width / 2;

    /* soh-cah-TOA
     * 
     * TOA = tan(0) = O / A
     *   T = tan( (horizontal FOV / 2) in radians )
     *   O = (frame width / 2) in mm
     *   A = depth in mm
     *   
     *   O = A * T
     */
 
    double T = Math.Tan((Math.PI * 180) / hFov);
    double pixelWidth = T * depth;

    return pixelWidth / numPixels;
}
Now that we know the length of each pixel, we can adjust the size of our head-tracking image to be a consistent “length”. The dimensions of the image will change as I move but the amount of space around my head remains consistent. The following calculates a 50 cm (~19”) image around the tracked position of my head:
double imageSize = 500 / CalculatePixelWidth(depthDesc, depth);

int x = (int)(Math.Floor(colorPoint.X + 0.5) - (imageSize / 2));
int y = (int)(Math.Floor(colorPoint.Y + 0.5) - (imageSize / 2));

WriteableBitmap faceImage = _bmp.Crop(new Rect(x,y, imageSize, imageSize));
Happy Coding.

Monday, March 24, 2014

Drawing Kinect V2 Body Joints

So far the posts in my Kinect for Windows v2 series have concentrated on the Depth, Color and BodyIndex data sources. Today I want to highlight how to access the Body data stream.
ColorExample-02-18-51
This is an early preview of the new Kinect for Windows, so the device, software and documentation are all preliminary and subject to change.
The new Kinect sensor and SDK has improved skeleton tracking so significantly that the Microsoft team has changed the name of their Skeleton class to Body. In addition to being able to track six skeletons (instead of 2), the API is shaping up to include features for Activities (facial features such as left eye open), Expressions (happy or neutral), Appearance (wearing glasses), Leaning (left or right) and the ability to track if the user is actively looking at the sensor. I’ll dive into those in up coming posts, but for now I want to focus on Joint information.
Generally speaking, the body tracking capability and joint positions are improved over previous version of the SDK. Specifically, the positions for hips and shoulders are more accurate. Plus version 2 has introduced 5 new joints (Neck, Hand-Tip-Left, Thumb-Left, Hand-Tip-Right, Thumb-Right) bringing the total number of joints to 25.
Body-Joints
Getting the Body data from the sensor is slightly different than the other data streams. First you must initialize an array of Body with a specific size and then pass it into the GetAndRefreshBodyData method on the BodyFrame. The SDK uses a memory saving optimization that updates the items in this array rather than creating a new set each time. So if you want to hang onto an instance of Body between frames, you need to copy it to another variable and replace the item in the Array with a null value.
The following shows how setup and populate an array of Body objects per frame:

private void SetupCamera()
{
    _sensor = Sensor.Default;
    _sensor.Open();

    _bodies = new Body[_sensor.BodySouceFrame.BodyCount];

    _reader = _sensor.BodySourceFrame.OpenReader();
    _reader.FrameArrived += FrameArrived;
}

private void FrameArrived(object sender, BodyFrameArrivedEventArgs e)
{
    using (BodyFrame frame = e.Frame.AcquireFrame())
    {
        if (frame == null)
            return;

        frame.GetAndRefreshBodyData(_bodies);
    }
}
Once you have a Body to work with, the joints and other features are provided to us as dictionaries which allows us to access the joints by name. For example:
Joint head = body.Joints[JointType.Head];
Each Body and Joint is also equipped with tracking confidence. Given this information, we can loop through the joints and only display the items that we know are actively being tracked. The following example iterates over the Body and Joint collections, and uses the CoordinateMapper to translate the joint position to dimensions on our color image. For simplicity sake, I’m simply coloring the surrounding pixels to illustrate their position.
private void DrawBodies(BodyFrame bodyFrame, int colorWidth, int colorHeight)

    bodyFrame.GetAndRefreshBodyData(_bodies);

    foreach(Body body in _bodies)
    {
        if (!body.IsTracked)
            continue;

        IReadOnlyDictionary<JointType, Joint> joints = body.Joints;

        var jointPoints = new Dictionary<JointType, Point>();

        foreach(JointType jointType in joints.Keys)
        {
            Joint joint = joints[jointType];
            if (joint.TrackingState == TrackingState.Tracked)
            {
                ColorSpacePoint csp = _coordinateMapper.MapCameraPointToColorSpace(joint.Position);
                jointPoints[jointType] = new Point(csp.X, csp.Y);                                                                        
            }
        }

        foreach(Point point in jointPoints.Values)
        {
            DrawJoint(ref _colorData, point, colorWidth, colorHeight, 10);
        }
    }
}

private void DrawJoint(ref byte[] colorData, Point point, int colorWidth, int colorHeight, int size = 10)
{
    int colorX = (int)Math.Floor(point.X + 0.5);
    int colorY = (int)Math.Floor(point.Y + 0.5);

    if (!IsWithinColorFrame(colorX, colorY, colorWidth, colorHeight))
        return;

    int halfSize = size/2;

    // loop through pixels around the point and make them red
    for (int x = colorX - halfSize; x < colorX + halfSize; x++)
    {
        for(int y = colorY - halfSize; y < colorY + halfSize; y++)
        {
            if (IsWithinColorFrame(x,y, colorWidth, colorHeight))
            {
                int index = ((colorWidth * y) + x) * bytesPerPixel;
                colorData[index + 0] = 0;
                colorData[index + 1] = 0;
                colorData[index + 2] = 255;
            }
        }
    }
}

private bool IsWithinColorFrame(int x, int y, int width, int height)
{
    return (x >=0 && x < width && y >=0 && y < height);
}
Happy Coding.

Wednesday, March 19, 2014

Mapping between Kinect Color and Depth

Continuing my series of blog posts about the new Kinect v2, I’d like to build upon my last post about the HD color stream with some of the depth frame concepts that I've used in the last few posts. Unlike the previous version of the Kinect, the Kinect v2 depth stream is not the same dimensions as the new color stream. This post will illustrate how the two are related and how you can map depth data to the color stream using the CoordinateMapper.
This is an early preview of the new Kinect for Windows, so the device, software and documentation are all preliminary and subject to change.
While the Kinect’s color and depth streams are represented as arrays of information, you simply cannot compare the x & y coordinates between the sets of data equally: the bytes of the color frame represent color pixels from the color camera; the bits of the depth frame represent Cartesian distance from the depth camera. Fortunately, the Kinect SDK ships with a mapping utility that can convert data between the different “spaces”.
For this post I’m going to use the MultiFrameSourceReader to access both the depth and color frames at the same time and we’ll use the CoordinateMapper.MapDepthFrameToColorSpace method to project the depth data into an array of ColorSpacePoint. Here’s the skeleton for processing frames as they arrive:

private void FrameArrived(object sender, MultiSourceFrameArrivedEventArgs e) 
{ 
    var reference = e.FrameReference; 

    MultiSourceFrame multiSourceFrame = null;
    ColorFrame colorFrame = null; 
    DepthFrame depthFrame = null; 

    try 
    { 
        using (_frameCounter.Increment()) 
        { 
            multiSourceFrame = reference.AcquireFrame(); 
            if (multiSourceFrame == null) 
                return; 

            using (multiSourceFrame) 
            { 
                colorFrame = multiSourceFrame.ColorFrameReference.AcquireFrame();
                depthFrame = multiSourceFrame.DepthFrameReference.AcquireFrame(); 

                if (colorFrame == null | depthFrame == null) 
                    return; 

                // initialize color frame data 
                var colorDesc = colorFrame.FrameDescription; 
                int colorWidth = colorDesc.Width; 
                int colorHeight = colorDesc.Height; 

                if (_colorFrameData == null) 
                { 
                    int size = colorDesc.Width * colorDesc.Height; 
                    _colorFrameData = new byte[size * bytesPerPixel]; 
                } 

                // initialize depth frame data 
                var depthDesc = depthFrame.FrameDescription; 

                if (_depthData == null) 
                { 
                    uint depthSize = depthDesc.LengthInPixels; 
                    _depthData = new ushort[depthSize]; 
                    _colorSpacePoints = new ColorSpacePoint[depthSize]; 
                } 

                // load color frame into byte[] 
                colorFrame.CopyConvertedFrameDataToArray(_colorFrameData, ColorImageFormat.Bgra); 

                // load depth frame into ushort[] 
                depthFrame.CopyFrameDataToArray(_depthData); 

                // map ushort[] to ColorSpacePoint[] 
                _sensor.CoordinateMapper.MapDepthFrameToColorSpace(_depthData, _colorSpacePoints); 

                // TODO: do something interesting with depth frame 

                // render color frame 
                _bmp.WritePixels( 
                    new Int32Rect(0, 0, colorDesc.Width, colorDesc.Height), 
                    _colorFrameData, 
                    colorDesc.Width * bytesPerPixel, 
                    0); 
            } 
        } 
    } 
    catch { } 
    finally 
    { 
        if (colorFrame != null) 
            colorFrame.Dispose(); 

        if (depthFrame != null) 
            depthFrame.Dispose(); 

    }
}
The MapDepthFrameToColorSpace method copies the depth data into an array of ColorSpacePoint where each item in the array corresponds to the items in the depth data. We can use the X & Y coordinates of the ColorSpacePoint to find the color data as demonstrated below. There’s one caveat: not all points in the depth array contain data that can be mapped to color pixels. Some points might be too close or too far, or there’s no depth data because it’s a shadow or reflective material.
The following snippet shows us how to locate the color bytes from the ColorSpacePoint:
// we need a starting point, let's pick 0 for now
int index = 0;

ushort depth = _depthData[index];
ColorSpacePoint point = _colorSpacePoints[index];

// round down to the nearest pixel
int colorX = (int)Math.Floor(point.X + 0.5);
int colorY = (int)Math.Floor(point.Y + 0.5);

// make sure the pixel is part of the image
if ((colorX >= 0 && (colorX < colorWidth) && (colorY >= 0) && (colorY < colorHeight))
{

    int colorImageIndex = ((colorWidth * colorY) + colorX) * bytesPerPixel;

    byte b = _colorFrameData[colorImageIndex];
    byte g = _colorFrameData[colorImageIndex + 1];
    byte r = _colorFrameData[colorImageIndex + 2];
    byte a = _colorFrameData[colorImageIndex + 3];

}
If we loop through the depth data and use the above technique we can draw our depth data on top of our color frame. For this image, I’m drawing the depth data using an intensity technique described in an earlier post. It looks like this:
ColorExample-10-16-26
You may notice that the pixels are far apart and don’t go to the entire edge of the image. This makes sense because our depth data is a smaller resolution (424 x 512 compared to 1080 x 1920) and tighter viewing angle (70.6° compared to 84°). The mapping also isn’t perfect in this release (remember this is a developer preview!)
We can use the same technique to draw each pixel of the depth frame using values from the color frame, like so:
// clear the pixels before we color them
Array.Clear(_pixels, 0, _pixels.Length);

for (int depthIndex = 0; depthIndex < _depthData.Length; ++depthIndex)
{
    ColorSpacePoint point = _colorSpacePoints[depthIndex];

    int colorX = (int)Math.Floor(point.X + 0.5);
    int colorY = (int)Math.Floor(point.Y + 0.5);
    if ((colorX >= 0) && (colorX < colorWidth) && (colorY >= 0) && (colorY < colorHeight))
    {
        int colorImageIndex = ((colorWidth * colorY) + colorX) * bytesPerPixel;
        int depthPixel = depthIndex * bytesPerPixel;

        _pixels[depthPixel] = _colorData[colorImageIndex];
        _pixels[depthPixel + 1] = _colorData[colorImageIndex + 1];
        _pixels[depthPixel + 2] = _colorData[colorImageIndex + 2];
        _pixels[depthPixel + 3] = 255;
    }
}
…which results in the following image:
ColorExample-10-09-28
So as you can see, we can easily map between the two coordinate spaces. I intended to build upon this further in upcoming posts, so if you haven’t already, add me to your favorite RSS reader.
Happy coding.

Tuesday, March 18, 2014

Accessing Kinect Color Data

One of the new features of the Kinect v2 is the upgraded color camera which now provides images with 1920 x 1080 resolution. This post will illustrate how you can get access to this data stream.

This is an early preview of the new Kinect for Windows, so the device, software and documentation are all preliminary and subject to change.

Much like the other data streams, accessing the color image is done using a reader:

private void SetupCamera();

    var sensor = KinectSensor.Default;
    sensor.Open();
    ColorFrameReader reader = sensor.ColorFrameSource.OpenReader();
    reader.FrameArrived += ColorFrameArrived;
}

In this example, I'm simply copying the frames to a WriteableBitmap which is bound to the view.

private readonly int bytesPerPixel = (PixelFormats.Bgr32.BitsPerPixel + 7) / 8;
private readonly WriteableBitmap _bmp = new WriteableBitmap(1920, 1080, 96, 96, PixelFormats.Bgra32, null);
Byte[] _frameData = null;

private void ColorFrameArrived(object sender, ColorFrameArrivedEventArgs e)
{
    var reference = e.FrameReference;

    try
    {
        var frame = reference.AcquireFrame();
        if (frame == null) return;

        using(frame)
        {
            FrameDescription desc = frame.FrameDescription;
            var size = desc.Width * desc.Height;
            
            if (_frameData == null)
            {
                _bmp = new WriteableBitmap(desc.Width, desc.Height, 96, 96, PixelFormats.Bgr32, null);
                _frameData = new byte[size * bytesPerPixel];
            }

            frame.CopyConvertedFrameDataToArray(_frameData, ColorImageFormat.Bgra);

            _bmp.WritePixels(
                new Int32Rect(0, 0, desc.Width, desc.Height),
                _frameData,
                desc.Width * bytesPerPixel,
                0);
        }
    }
    catch
    {}

}

As you’d expect, this updates the view every 30 frames per second with a HD image. The wide-angle lens of the camera (84°) captures a fair amount of my office.

ColorExample-09-41-17

Happy Coding.

Monday, February 10, 2014

PhotoBooth (Free Code)

I’ve been blogging recently about some of my experiments with the Kinect for Windows v2 developer preview. Most of my experiments have been based on the supplied samples, with a lot of copy and paste action. I’ve pulled some of the repeated code into some re-usable classes.

Here’s some code that I’ve moved out of the samples into a reusable component that takes a screen shot of the Kinect output. As an added bonus, I’ve added a count-down timer. The code provided here is based on the original samples, modified slightly. Free as in beer.

Here’s an example of how to use it.

First, add the visual for the count-down timer into your UI. I’m putting mine below the image that shows my Kinect output.

<ViewBox>
    <Image Source="{Binding ImageSource}" Stretch="UniformToFill" />
    <TextBlock 
        Text="{Binding CountDownTimer}"
        HorizontalAlignment="Center"
        VerticalAlignment="Center"
        FontSize="96"
        Foreground="White" />
</ViewBox>

<Button 
    Style="{StaticResource ScreenshotButton}"
    Content="Screenshot"
    Click="ScreenshotButton_Click" />

Next, initialize the PhotoBooth in your ViewModel. Note that I’m being very lazy and I’m using the code-behind of the XAML as my ViewModel. You know I’m not a big fan of this, I’m just following the lead of the Kinect examples so let’s pretend I didn’t do that. Do proper MVVM, kids.

public class MainWindow : Window, INotifyPropertyChanged
{
    private KinectSensor _sensor;
    private DepthFrameReader _reader;
    private WriteableBitmapImage _bmp;

    // see my last post on how to use the framecounter!
    private FrameCounter _frameCounter;
    private PhotoBooth _photoBooth;

    private string _statusText;
    private string _countDownTimer;


    public MainWindow()
    {
        InitializeComponent();

        _sensor = KinectSensor.Default;
        _sensor.Open();
        _reader = _sensor.DepthFrameSource.OpenReader();
        _reader.FrameArrived += FrameArrived;

        _photoBooth = new PhotoBooth();
        _photoBooth.PropertyChanged += (o,e) => 
            CountDownTimer = _photoBooth.TimeDisplay;       

        _frameCounter = new FrameCounter();
        _frameCounter.PropertyChanged += (o,e) =>
            StatusText = String.Format("{0} FPS", _frameCounter.FramesPerSecond);

        DataContext = this;
    }

    public string StatusText { /* get/set property changed omitted */ }
    public string CountDownTimer { /* get/set property changed omitted */ }
    public BitmapImage ImageSource { /* get/set property changed omitted */ }

    async void ScreenshotButton_Click(object sender, EventArgs e)
    {

        string fileName = await _photoBooth.TakePhoto(_bmp);

        if (!String.IsNullOrEmpty(fileName)
        {
            StatusText = string.Format("Snapshot created: {0}", fileName);
        }
        else
        {
            StatusText = "Couldn't create snapshot :(";
        }

        // allow our message to show for 5 seconds...
        _frameCounter.DeferFrameCount(TimeSpan.FromSeconds(5));
    }

    void FrameArrived(object sender, DepthFrameSourceEventArgs e)
    {
        // create bitmap, write depth data to bitmap
    }
}

By default, the Photobooth will create a new PNG in your Pictures folder with a random guid as the file name. You can change both of these to suit your needs.

// default directory is My Pictures
_photoBooth.BaseDirectory = "C:\Snaps";

// specify the file naming convention. will add .png to the end...
_photoBooth.CreateFileName = () => {
    string time = System.DateTime.Now.ToString("hh'-'mm'-'ss", CultureInfo.CurrentUICulture.DateTimeFormat);
    return String.Format("KinectScreenshot-PhotoBoothSample-{0}", time);    
};

Click to expand to get the full source.

public class PhotoBooth : INotifyPropertyChanged
{
    private int _timeLeft;
    private Func<string> _createFileName;
    private string _baseDirectory;
    private TimeSpan _waitInterval;

    public PhotoBooth()
    {
        BaseDirectory = Environment.GetFolderPath(Environment.SpecialFolder.MyPictures);
        WaitInterval = TimeSpan.FromSeconds(5);
    }

    public string BaseDirectory
    {
        get { return _baseDirectory; }
        set { _baseDirectory = value; }
    }

    public TimeSpan WaitInterval
    {
        get { return _waitInterval; }
        set { _waitInterval = value; }
    }

    public string TimeDisplay
    {
        get
        {
            if (TimeLeft > 0)
            {
                return TimeLeft.ToString();
            }

            return String.Empty;
        }
    }

    public int TimeLeft
    {
        get { return _timeLeft; }
        protected set
        {
            if (_timeLeft != value)
            {
                _timeLeft = value;
                var handler = PropertyChanged;
                if (handler != null)
                {
                    handler(this, new PropertyChangedEventArgs("TimeLeft"));
                    handler(this, new PropertyChangedEventArgs("TimeDisplay"));
                }
            }
        }
    }

    public async Task<string> TakePhoto(WriteableBitmap bitmap, string path = null, int seconds = -1)
    {
        if (string.IsNullOrEmpty(path))
        {
            path = BaseDirectory;
        }

        if (seconds == -1)
        {
            seconds = (int)WaitInterval.TotalSeconds;
        }

        if (seconds > 0)
        {
            TimeLeft = seconds;
            while (TimeLeft > 0)
            {
                await Task.Delay(1000);
                TimeLeft = TimeLeft - 1;
            }
        }

        string fileName = Path.Combine(path, CreateFileName() + ".png");

        bool success = SaveBitmap(fileName, bitmap);

        return success ? fileName : String.Empty;
    }

    public Func<string> CreateFileName
    {
        get
        {
            if (_createFileName == null)
            {
                _createFileName = () => Guid.NewGuid().ToString();
            }
            return _createFileName;
        }
        set
        {
            _createFileName = value;
        }
    }

    public event PropertyChangedEventHandler PropertyChanged;

    private bool SaveBitmap(string path, BitmapSource bitmapSource)
    {
        if (bitmapSource == null)
            return false;

        // create a png bitmap encoder which knows how to save a .png file
        BitmapEncoder encoder = new PngBitmapEncoder();

        // create frame from the writable bitmap and add to encoder
        encoder.Frames.Add(BitmapFrame.Create(bitmapSource));


        // write the new file to disk
        try
        {
            // FileStream is IDisposable
            using (FileStream fs = new FileStream(path, FileMode.Create))
            {
                encoder.Save(fs);
            }

        }
        catch (IOException)
        {
            return false;
        }

        return true;            
    }
}

Happy coding

submit to reddit

Thursday, February 06, 2014

FPS and CPU Counter (Free Code)

I’ve been blogging recently about some of my experiments with the Kinect for Windows v2 developer preview. Most of my experiments have been based on the supplied samples, with a lot of copy and paste action. I’ve pulled some of the repeated code into some re-usable classes.

Here’s some code that I’ve moved out of the samples into a reusable component that measures frames-per-second and the amount of time taken per frame. The code provided here is based on the original samples, modified slightly. Free as in beer, though I doubt any of you will buy me beer so let’s just say “free”.

Example how to use:

public class MainWindow : Window, INotifyPropertyChanged
{
    private FrameCounter _frameCounter;
    private KinectSensor _sensor;
    private DepthFrameReader _reader;


    public MainWindow()
    {
        InitializeComponent();

        _frameCounter = new FrameCounter();
        _frameCounter.PropertyChanged += (o,e) =>
            StatusText = String.Format("FPS = {0:N1} / CPU = {1:N6}",
                            _frameCounter.FramesPerSecond,
                            _frameCounter.CpuTimePerFrame);

        _sensor = KinectSensor.Default;
        _sensor.Open();
        _reader = sensor.DepthFrameSource.OpenReader();
        _reader.FrameArrived += FrameArrived;

        this.DataContext = this;
    }

    private void FrameArrived(object sender, DepthFrameArrivedEventArgs e)
    {
        var reference = e.FrameReference;
        DepthFrame depthFrame = null;

        try
        {
            // increment the frame counter
            using (_frameCounter.Increment())
            {
                depthFrame = reference.AcquireFrame();
                
                // do work with depth data...


            } // disposing the frame counter will calculate CPU time for this block
        }
        catch
        {
        }
        finally
        {
            if (depthFrame != null)
                depthFrame.Dispose();
        }
    }

}

Expand the block below for code goodness.

public class FrameCounter : INotifyPropertyChanged
{
    private Stopwatch _stopWatch;
    private uint _framesSinceUpdate = 0;
    private DateTime _nextStatusUpdate = DateTime.MinValue;
    private CpuCounter _cpuCounter;
    private double _fps;
    private double _cpuTime;

    public FrameCounter()
    {
        _stopWatch = new Stopwatch();
        _cpuCounter = new CpuCounter(this);
    }

    public double FramesPerSecond
    {
        get { return _fps; }
        protected set
        {
            if (_fps != value)
            {
                _fps = value;
                NotifyPropertyChanged("FramesPerSecond");
            }
        }
    }

    public double CpuTimePerFrame
    {
        get { return _cpuTime; }
        protected set
        {
            if (_cpuTime != value)
            {
                _cpuTime = value;
                NotifyPropertyChanged("CpuTimePerFrame");
            }
        }
    }

    public event PropertyChangedEventHandler PropertyChanged;

    public void DeferFrameCount(TimeSpan timeToWait)
    {
        this._nextStatusUpdate = DateTime.Now + timeToWait;
    }

    public IDisposable Increment()
    {
        this._framesSinceUpdate++;

        if (DateTime.Now >= this._nextStatusUpdate)
        {
            double fps = 0.0;
            double cpuTime = 0.0;

            if (this._stopWatch.IsRunning)
            {
                this._stopWatch.Stop();
                fps = this._framesSinceUpdate / this._stopWatch.Elapsed.TotalSeconds;
                cpuTime = this._cpuTime / fps;
                this._stopWatch.Reset();
            }

            this._nextStatusUpdate = DateTime.Now + TimeSpan.FromSeconds(1);
            this.FramesPerSecond = fps;
            this.CpuTimePerFrame = cpuTime;
        }

        if (!this._stopWatch.IsRunning)
        {
            this._framesSinceUpdate = 0;
            this._cpuTime = 0;
            this._stopWatch.Start();
        }

        _cpuCounter.Reset();
        return _cpuCounter;
    }

    internal void IncrementCpuTime(double amount)
    {
        _cpuTime += amount;
    }

    public void Reset()
    {
        _nextStatusUpdate = DateTime.MinValue;
        _framesSinceUpdate = 0;
        FramesPerSecond = 0;
        CpuTimePerFrame = 0;
    }

    protected void NotifyPropertyChanged(string propertyName)
    {
        if (PropertyChanged != null)
        {
            PropertyChanged(this, new PropertyChangedEventArgs(propertyName));
        }
    }

    public class CpuCounter : IDisposable
    {
        FrameCounter _parent;
        Stopwatch _stopWatch;

        internal CpuCounter(FrameCounter parent)
        {
            _parent = parent;
            _stopWatch = new Stopwatch();
        }

        public TimeSpan Elapsed
        {
            get
            {
                return _stopWatch.Elapsed;
            }
        }

        public void Reset()
        {
            _stopWatch.Reset();
            if (!_stopWatch.IsRunning)
            {
                _stopWatch.Start();
            }
            
        }

        public void Dispose()
        {
            _stopWatch.Stop();
            _parent.IncrementCpuTime(_stopWatch.Elapsed.TotalMilliseconds);
        }
    }
}

Happy coding.

submit to reddit

Tuesday, February 04, 2014

Comparing Kinect v1 and v2 Depth Data

In this post I show how the new Kinect v2 stacks up against the previous sensor. I’ve been blogging about the new Kinect as part of the Kinect for Windows Developer Preview. First and foremost, I have to get this legal disclaimer out of the way…

This is an early preview of the new Kinect for Windows, so the device, software and documentation are all preliminary and subject to change.

As part of this post, I thought it would be fun to build an application that targets both versions of the device and toggle back and forth between them for comparison sake. There’s a little bit of magic involved as both have the same namespaces and assembly name. Here’s you how you can do the same.

  1. Make a copy of the v1.8 Microsoft.Kinect.dll as Microsoft.Kinect.v1.dll
  2. In your project, add a reference this assembly and change the alias from global to v1.
  3. Do the same for v2.
  4. Add the following code above your using statements.
extern alias v1;
extern alias v2;

using KinectV1 = v1::Microsoft.Kinect;
using KinectV2 = v2::Microsoft.Kinect;

What’s changed between versions?

The first and perhaps most surprising difference is that the frame resolution of the depth data is 512 x 424. Don’t panic. Yes, the Kinect v1 supports a higher resolution of 640x480 resolution, but don’t get caught up on that. This apparently is the native resolution of the camera and although Microsoft may expose other resolutions in a later release, the differences between Time of Flight and Structured Light more than make up for this difference.

As I understand it, Structured Light (the depth technology in v1) reconstructs the depth data by analyzing deformations of a projected IR pattern. This produces approximations for pixels between the projected points. Time of Flight (used in Kinect v2) however blasts the scene with IR and measures how quickly the light comes back to the sensor. This produces better results as each pixel has a unique value. Time of Flight sensors also tend to have lower resolutions, so 512 x 424 is actually pretty impressive compared to other TOF sensors.

The second main change is the field of view and focal length for the camera. The Kinect can see more of the area and can detect objects much closer.

Side by Side Comparisons

The following table shows the observable differences in the depth frame data between versions.

  Kinect V1 Kinect V2
Height 480 424
Width 640 512
Aspect Ratio 4:3 6:5
Horizontal FOV 58.5 70.6
Vertical FOV 45.6 60
Diagonal FOV 70 89.5
Min Depth 0.8 m 0.5 m
Max Depth 4.0 m 4.5 m

The following snapshots are taken from two Kinect devices sitting side by side:

WP_20140203_001

Kinect V1

As part of the structured light algorithm, parts of the depth data are approximated. The subject must be at least 800mm from the camera.

KinectScreenshot-ColorMap-09-27-48

Kinect V2

The new Time of Flight sensor provides greater depth qualities per pixel and a much improved signal to noise ratio despite having an odd 6:5 aspect ratio (512 x 424 px). The increased field of view allows the camera to see more of the scene.

KinectScreenshot-ColorMap-09-28-02

submit to reddit

Monday, January 27, 2014

Changing the way .net looks for dependencies

Whenever I have to clean up my machine in preparation for changing my hardware or an OS refresh, I always find some forgotten gems in the process. Here’s one that I’m throwing up onto my blog for prosperity sake.

Every so often I have a project with lots of assemblies and dependencies which results in a deployment folder that can be daunting to the uninitiated. It would be great if the deployment folder had the exe and the config file and all the other dependencies were hidden in subfolders.

Here’s an example of how to do that.

  1. First, create two projects. Shell is an exe and Dependency is a ClassLibrary
  2. Change the Build folder of Shell to point to “..\AssemblyOutput”
  3. Change the Build folder of Dependency to point to “..\AssemblyOutput\modules”
  4. In the Shell project, add a reference to the Dependency project
  5. In the Shell’s references, set the CopyLocal property of the Dependency reference to “False”. If you don’t do this, the assembly will be copied to the root folder of the Shell’s output folder.
  6. Add the following to the Shell’s app config:
<configuration>

  <runtime>
    <assemblyBinding xmlns="urn:schemas-microsoft-com:asm.v1">
      
      <!-- Probing path is relative to the executable. Multiple values are supported (bin;bin2;modules;modules\subfolder)-->
      <probing privatePath="modules"/>

    </assemblyBinding>
  </runtime>

</configuration>

Now when you run the project, the first time the code comes across a reference to a type inside the Dependency assembly, it will load the assembly in the modules folder.

Happy coding.

submit to reddit

Tuesday, January 14, 2014

Leverage the Kinect v2 underlying depth buffer

One of the great new features in the API for new Kinect for Windows v2 developer preview is the ability to access the underlying buffer for the data sources. Normally, you can get by with using the CopyFrameDataToArray method but this option allows you to get a little closer to the metal and possibly squeeze out a bit more performance. Frankly, this is a good thing since we’re iterating over 217,088 pixels at a rate of 30 times per second – anything helps.

This is an early preview of the new Kinect for Windows, so the device, software and documentation are all preliminary and subject to change.

Also note, because C# does not support pointer arithmetic by default, you must mark your methods with the unsafe keyword.

To access the buffer, we use the AccessUnderlyingBuffer method on the DepthFrame which tells us how big the size of the buffer is and a pointer to the buffer.

unsafe
{
    uint size = 0;
    IntPtr ptr = IntPtr.Zero;

    depthFrame.AccessUnderlyingBuffer(out size, out ptr);

    // ptr now points to the buffer
}

Once we have a pointer to the buffer we can iterate through it’s contents. It’s up to us to understand how to parse the buffer and it’s not type-safe. We assume that the buffer is comprised of an array of ushort.

// this is also unsafe, so keep in the unsafe block above...
    
// obtain a pointer to the ushort array
ushort* data = (ushort*)ptr.ToPointer();

for(int i = 0; i < size; i++)
{
    ushort depth = data[i];

    // work with depth data
}

In my modest experimentation, switching to the buffer reduced cpu time between 0.5 – 1.0 milliseconds. Your mileage will obviously vary. Here’s a small clip of me toggling between the two approaches. The CPU counter in the bottom left is averaged over the number of frames per second and includes the time to render and update the image. There’s a subtle difference.

Happy Coding!

submit to reddit

Friday, January 10, 2014

Finding Pixels for Bodies in Kinect for Windows v2

In this post, I’m using Depth and BodyIndex data to create a depth visualization that highlights the active users.

This is an early preview of the new Kinect for Windows, so the device, software and documentation are all preliminary and subject to change.

KinectScreenshot-ColorMap-09-29-48

This post illustrates a few breaking changes from v1 of the Kinect for Windows API.

First off, the event signature to receive all data from the sensor has changed. Rather than getting data from the AllFramesReady event of the sensor, version 2 now uses a MultiSourceFrameReader that has a MultiSourceFrameArrived event. The MultiSourceFrameReader allows you to specify which data streams you wish to receive. This is a relatively minor change.

The other minor change this post illustrates is that the depth data no longer contains a byte to indicate the player index. Rather than applying a bitmask to the depth data, it is now exposed as its own data stream.

this.sensor = KinectSensor.Default;
this.sensor.Open();
this.reader = sensor.OpenMultiFrameSourceReader(
                        FrameSourceTypes.BodyIndex |
                        FrameSourceTypes.Depth
                        );
this.reader.MultiSourceFrameArrived += MultiSourceFrameArrived;

DepthFrameDescription desc = sensor.DepthFrameSource.FrameDescription;

this.bytesPerPixel = (PixelFormats.Bgr32.BitsPerPixel + 7) / 8;
this.depthFrameData = new ushort[desc.Width * desc.Height];
this.bodyIndexData = new byte[desc.Width * desc.Height];

this.pixels = new byte[desc.Width * desc.Height * this.bytesPerPixel];
this.bitmap = new WriteableBitMap(desc.Width, desc.Height, 96, 96, PixelFormat.Bgr32, null);

When the frame arrives, I fill both the depth and body index arrays with the available data. All that's left is to loop through the depth data and color the pixels. In this case, I'm using the GradientColorMap from my last post to decorate color pixels differently than background pixels.

void MultiSourceFrameArrived(object sender, MultiSourceFrameArrivedEventArgs e)
{
    MultiSourceFrameReference frameReference = e.FrameReference;

    MutliSourceFrame multiSourceFrame = null;
    DepthFrame depthFrame = null;
    BodyIndexFrame bodyIndexFrame = null;

    try
    {
        multiSourceFrame = frameReference.AcquireFrame();
        if (multiSourceFrame != null)
        {
            var depthFrameReference = multiSourceFrame.DepthFrameReference;
            var bodyIndexFrameReference = multiSourceFrame.BodyIndexFrameReference;

            depthFrame = depthFrameReference.AcquireFrame();
            bodyIndexFrame = bodyIndexFrameReference.AcquireFrame();

            if ((depthFrame != null) && (bodyIndexFrame != null))
            {
                depthFrame.CopyFrameDataToArray(this.depthFrameData);
                bodyIndexFrame.CopyFrameDataToArray(this.bodyIndexFrameData);

                int pixelIndex = 0;
                for (int i = 0; i < depthFrameData.Length; ++i)
                {
                    ushort depth = depthFrameData[i];
                    byte player = this.bodyIndexFrameData[i];
                    Color pixelColor;

                    if (player == 0xff) // nobody here
                    {
                        pixelColor = this.grayscaleMap.GetColorForDepth(depth);
                    }
                    else
                    {
                        pixelcolor = this.playerMap.GetColorForDepth(depth);
                    }
                }
                
                pixels[pixelIndex++] = pixelColor.B;
                pixels[pixelIndex++] = pixelColor.G;
                pixels[pixelIndex++] = pixelColor.R;
                pixelIndex++;
             }

             this.bitmap.WritePixels(
                 new Int32Rect(0, 0, depthWidth, depthHeight),
                 this.pixels,
                 depthWidth * this.cbytesPerPixel,
                 0);
            }

        }

    }
    catch
    {
    }
    finally
    {
        // MultiSourceFrame, DepthFrame, ColorFrame, BodyIndexFrame are IDispoable
        if (depthFrame != null)
        {
           depthFrame.Dispose();
           depthFrame = null;
        }

        if (bodyIndexFrame != null)
        {
            bodyIndexFrame.Dispose();
            bodyIndexFrame = null;
        }

        if (multiSourceFrame != null)
        {
            multiSourceFrame.Dispose();
            multiSourceFrame = null;
        }
    }

}

Easy peasy! Happy Coding!

Wednesday, January 08, 2014

Accessing Depth data with Kinect v2

This is an early preview of the new Kinect for Windows, so the device, software and documentation are all preliminary and subject to change.

Accessing Depth Data for the new Kinect for Windows API v2 is easy and very similar to the previous version of the API. We’ll do some more with this in a bit, but here’s a code snippet that’s been paraphrased from the v2 samples:

// Get the Sensor
this.sensor = KinectSensor.Default;

if (this.sensor! = null)
{
    // If we have a sensor, open it
    this.sensor.Open();

    // this example will use a BGR format image...    
    this.bitsPerPixel = (PixelFormats.Bgr32.BitsPerPixel + 7) / 8;

    // initialize some arrays based on the depth frame size
    FrameDescription desc = this.sensor.DepthFrameSource.FrameDescription;
    int size = desc.Width * desc.Height;
    this.frameData = new ushort[size];
    this.pixels = new byte[size * bitsPerPixel];

    this.bitmap = new WriteableBitmap(desc.Width, desc.Height, 96.0, 96.0, PixelFormats.Bgra32, null);
    
    // obtain a reader and subscribe to events
    this.reader = this.sensor.DepthFrameSource.OpenReader();
    this.reader.FrameArrived += FrameArrived;
}


void FrameArrived(object sender, DepthFrameArrivedEventArgs e)
{
    DepthFrameReference frameReference = e.FrameReference;

    try
    {
        DepthFrame frame = frameReference.AcquireFrame();

        if (frame != null)
        {
            // frame is disposable
            using(frame)
            {

                var desc = frame.FrameDescription;

                frame.CopyFrameDataToArray(this.frameData);

                int colorPixelIndex = 0;

                // loop through the frame data
                for(int i = 0; this.frameData.Length; ++i)
                {
                    ushort depth = this.frameData[i];

                    // do something interesting with depth data here
                    // see examples below...
                    // Color color = this.colorMap.GetColorForDepth(depth);
                    Color color = new Color();

                    // assign values to individual pixels
                    // BGRA format
                    this.pixels[colorPixelIndex++] = color.B; 
                    this.pixels[colorPixelIndex++] = color.G;
                    this.pixels[colorPixelIndex++] = color.R;
                    ++colorPixelIndex; // skip Alpha color
                }

                this.bitmap.WritePixels(
                    new Int32Rect(0, 0, desc.Width, desc.Height),
                    this.pixels,
                    desc.Width * this.bitsPerPixel,
                    0);

            }
        }
    }
    catch(Exception)
    {
        // frame might no longer be available
    }
}

Fairly straight-forward, right? The biggest difference from v1 is that we must access the data from an event on the reader rather than an event from the sensor.

Intensity

The example in the SDK truncates the depth data down to the last byte, which produces a grayscale effect that “wraps” every 255 millimeters. It doesn’t look very nice, but the key thing to observe is just how detailed the depth data is. If you look closely, you can clearly make out facial features and other fine details such as the lines in my hands.

KinectScreenshot-ColorMap-08-37-27

// paraphrased from the SDK examples..
// truncates the depth data down to the lowest byte, which
// produces an "intensity" value between 0-255

ushort minDepth = desc.DepthMinReliableDistance;
ushort maxDepth = desc.DepthMaxReliableDistance;

// To convert to a byte, we're discarding the most-significant
// rather than the least-significant bits.
// We're preserving detail, although the intensity will "wrap."
// Values outside the reliable depth range are mapped to 0 (black).
byte intensity = (byte)(depth >= minDepth && depth <= maxDepth) ? depth : 0;

// convert intensity to a color (to fit into our example above)
var color = new Color()
        {
            R = intensity,
            G = intensity,
            B = intensity
        };

Prettier examples

Here’s a few examples I’ve cooked up. From past experience with the Kinect v1 we observed that you can improve performance by reducing the amount of calculations per frame. One approach I’ve used it to pre-calculate the results into a hash table between 500 and 4500 milimeters. There are likely other optimizations that can be made, but this is easy.

For fun, I’ve created an abstraction called a DepthMap:

abstract class DepthMap
{
    protected ushort minDepth;
    protected ushort maxDepth;

    protected DepthMap(ushort minDepth, ushort maxDepth)
    {
        this.minDepth = minDepth;
        this.maxDepth = maxDepth;
    }

    public abstract Color GetColorForDepth(ushort depth);
}

HSV Color Wheel

Rather than a grayscale effect, I found a few great online samples on how to implement a HSV color wheel which produces this fun rainbow effect. I’m gradually transitioning between colors by looking at the most significant bits.

KinectScreenshot-ColorMap-08-47-20

double h, s, v;
int r, g, b;

h = ((depth - minDepth) / (maxDepth - minDepth)) * 360;
v = 1;
s = 1;

Hsv.HsvToRg(h, v, s, out r, out g, out b);

return new Color {
        R = (byte)r,
        G = (byte)g,
        B = (byte)b
    };

Color Gradients

The formula to produce a gradient is rather simple, you take a percentage of one color and a remaining percentage of another color. This color map allows me to pick the colors I want to use, which produces some interesting effects. Using an array of colors (Red, Yellow, Green, Yellow, Red) I can color code depth to reflect an ideal position from the camera.

KinectScreenshot-ColorMap-08-47-39

var colorMap = new GradientColorMap(
    minDepth, 
    maxDepth, 
    new[] { 
        Colors.Red, 
        Colors.Yellow, 
        Colors.Green, 
        Colors.Yellow, 
        Colors.Red 
        }
    );

I can also create a boring grayscale effect by just using Black and White.

var colorMap = new GradientColorMap(
    minDepth,
    maxDepth,
    new[] {
        Color.White,
        Color.Black
        }
    );

You can see some of them in action here:

Happy coding!

Monday, January 06, 2014

Multiple Apps can use Kinect for Windows v2

One of the new features of the Kinect for Windows API is that multiple applications can simultaneously access the sensor, as demonstrated by this video:

This is an early preview of the new Kinect for Windows, so the device, software and documentation are all preliminary and subject to change.

The lower right hand corner of this video shows a console application, KinectService, that must be running in order to run the samples provided. It’s unclear what role the service will play in the future releases of the SDK but I’d expect that it runs as Windows Service or headless process running from the system tray.

The Kinect API has changed to reflect this ability. It’s a minor change that will break some of your existing v1 code, but should be relatively easy to port to v2. Previous versions of the API allowed you to access the sensor data by subscribing to events on the sensor. The new version of the API introduces a reader pattern that separates the events from the sensor. Each reader has access to their own frames, which should in theory allow for more modular applications.

Here’s a quick sample:

// v1
KinectSensor sensor = KinectSensor.Sensors[0];
sensor.DepthStream.Enable(DepthImageFormat.Resolution640x480Fps30);
sensor.Start();
sensor.DepthFrameReady += OnDepthFrameReady;

// v2
KinectSensor sensor = KinectSensor.Default;
sensor.Open();
DepthFrameReader reader = sensor.DepthFrameSource.OpenReader();
reader.FrameArrived += OnDepthFrameArrived;

This change is significant because most devices are exclusively locked to the application that is using it. This can be a real pain when your app crashes but doesn’t release the handle of the device.

I can see many options for this functionality:

  • A support engineer can remotely troubleshoot the environment setup without having to close the primary application
  • A background process could take periodic snapshots to produce timelapse videos of usage of your application
  • Applications produced by different vendors could simulatenously run on the same machine: A desktop application could dim the monitor when you're not looking at it while a gesture application augments your windows experience...

What will you do with this feature?