Monday, February 10, 2014

PhotoBooth (Free Code)

I’ve been blogging recently about some of my experiments with the Kinect for Windows v2 developer preview. Most of my experiments have been based on the supplied samples, with a lot of copy and paste action. I’ve pulled some of the repeated code into some re-usable classes.

Here’s some code that I’ve moved out of the samples into a reusable component that takes a screen shot of the Kinect output. As an added bonus, I’ve added a count-down timer. The code provided here is based on the original samples, modified slightly. Free as in beer.

Here’s an example of how to use it.

First, add the visual for the count-down timer into your UI. I’m putting mine below the image that shows my Kinect output.

<ViewBox>
    <Image Source="{Binding ImageSource}" Stretch="UniformToFill" />
    <TextBlock 
        Text="{Binding CountDownTimer}"
        HorizontalAlignment="Center"
        VerticalAlignment="Center"
        FontSize="96"
        Foreground="White" />
</ViewBox>

<Button 
    Style="{StaticResource ScreenshotButton}"
    Content="Screenshot"
    Click="ScreenshotButton_Click" />

Next, initialize the PhotoBooth in your ViewModel. Note that I’m being very lazy and I’m using the code-behind of the XAML as my ViewModel. You know I’m not a big fan of this, I’m just following the lead of the Kinect examples so let’s pretend I didn’t do that. Do proper MVVM, kids.

public class MainWindow : Window, INotifyPropertyChanged
{
    private KinectSensor _sensor;
    private DepthFrameReader _reader;
    private WriteableBitmapImage _bmp;

    // see my last post on how to use the framecounter!
    private FrameCounter _frameCounter;
    private PhotoBooth _photoBooth;

    private string _statusText;
    private string _countDownTimer;


    public MainWindow()
    {
        InitializeComponent();

        _sensor = KinectSensor.Default;
        _sensor.Open();
        _reader = _sensor.DepthFrameSource.OpenReader();
        _reader.FrameArrived += FrameArrived;

        _photoBooth = new PhotoBooth();
        _photoBooth.PropertyChanged += (o,e) => 
            CountDownTimer = _photoBooth.TimeDisplay;       

        _frameCounter = new FrameCounter();
        _frameCounter.PropertyChanged += (o,e) =>
            StatusText = String.Format("{0} FPS", _frameCounter.FramesPerSecond);

        DataContext = this;
    }

    public string StatusText { /* get/set property changed omitted */ }
    public string CountDownTimer { /* get/set property changed omitted */ }
    public BitmapImage ImageSource { /* get/set property changed omitted */ }

    async void ScreenshotButton_Click(object sender, EventArgs e)
    {

        string fileName = await _photoBooth.TakePhoto(_bmp);

        if (!String.IsNullOrEmpty(fileName)
        {
            StatusText = string.Format("Snapshot created: {0}", fileName);
        }
        else
        {
            StatusText = "Couldn't create snapshot :(";
        }

        // allow our message to show for 5 seconds...
        _frameCounter.DeferFrameCount(TimeSpan.FromSeconds(5));
    }

    void FrameArrived(object sender, DepthFrameSourceEventArgs e)
    {
        // create bitmap, write depth data to bitmap
    }
}

By default, the Photobooth will create a new PNG in your Pictures folder with a random guid as the file name. You can change both of these to suit your needs.

// default directory is My Pictures
_photoBooth.BaseDirectory = "C:\Snaps";

// specify the file naming convention. will add .png to the end...
_photoBooth.CreateFileName = () => {
    string time = System.DateTime.Now.ToString("hh'-'mm'-'ss", CultureInfo.CurrentUICulture.DateTimeFormat);
    return String.Format("KinectScreenshot-PhotoBoothSample-{0}", time);    
};

Click to expand to get the full source.

public class PhotoBooth : INotifyPropertyChanged
{
    private int _timeLeft;
    private Func<string> _createFileName;
    private string _baseDirectory;
    private TimeSpan _waitInterval;

    public PhotoBooth()
    {
        BaseDirectory = Environment.GetFolderPath(Environment.SpecialFolder.MyPictures);
        WaitInterval = TimeSpan.FromSeconds(5);
    }

    public string BaseDirectory
    {
        get { return _baseDirectory; }
        set { _baseDirectory = value; }
    }

    public TimeSpan WaitInterval
    {
        get { return _waitInterval; }
        set { _waitInterval = value; }
    }

    public string TimeDisplay
    {
        get
        {
            if (TimeLeft > 0)
            {
                return TimeLeft.ToString();
            }

            return String.Empty;
        }
    }

    public int TimeLeft
    {
        get { return _timeLeft; }
        protected set
        {
            if (_timeLeft != value)
            {
                _timeLeft = value;
                var handler = PropertyChanged;
                if (handler != null)
                {
                    handler(this, new PropertyChangedEventArgs("TimeLeft"));
                    handler(this, new PropertyChangedEventArgs("TimeDisplay"));
                }
            }
        }
    }

    public async Task<string> TakePhoto(WriteableBitmap bitmap, string path = null, int seconds = -1)
    {
        if (string.IsNullOrEmpty(path))
        {
            path = BaseDirectory;
        }

        if (seconds == -1)
        {
            seconds = (int)WaitInterval.TotalSeconds;
        }

        if (seconds > 0)
        {
            TimeLeft = seconds;
            while (TimeLeft > 0)
            {
                await Task.Delay(1000);
                TimeLeft = TimeLeft - 1;
            }
        }

        string fileName = Path.Combine(path, CreateFileName() + ".png");

        bool success = SaveBitmap(fileName, bitmap);

        return success ? fileName : String.Empty;
    }

    public Func<string> CreateFileName
    {
        get
        {
            if (_createFileName == null)
            {
                _createFileName = () => Guid.NewGuid().ToString();
            }
            return _createFileName;
        }
        set
        {
            _createFileName = value;
        }
    }

    public event PropertyChangedEventHandler PropertyChanged;

    private bool SaveBitmap(string path, BitmapSource bitmapSource)
    {
        if (bitmapSource == null)
            return false;

        // create a png bitmap encoder which knows how to save a .png file
        BitmapEncoder encoder = new PngBitmapEncoder();

        // create frame from the writable bitmap and add to encoder
        encoder.Frames.Add(BitmapFrame.Create(bitmapSource));


        // write the new file to disk
        try
        {
            // FileStream is IDisposable
            using (FileStream fs = new FileStream(path, FileMode.Create))
            {
                encoder.Save(fs);
            }

        }
        catch (IOException)
        {
            return false;
        }

        return true;            
    }
}

Happy coding

submit to reddit

Thursday, February 06, 2014

FPS and CPU Counter (Free Code)

I’ve been blogging recently about some of my experiments with the Kinect for Windows v2 developer preview. Most of my experiments have been based on the supplied samples, with a lot of copy and paste action. I’ve pulled some of the repeated code into some re-usable classes.

Here’s some code that I’ve moved out of the samples into a reusable component that measures frames-per-second and the amount of time taken per frame. The code provided here is based on the original samples, modified slightly. Free as in beer, though I doubt any of you will buy me beer so let’s just say “free”.

Example how to use:

public class MainWindow : Window, INotifyPropertyChanged
{
    private FrameCounter _frameCounter;
    private KinectSensor _sensor;
    private DepthFrameReader _reader;


    public MainWindow()
    {
        InitializeComponent();

        _frameCounter = new FrameCounter();
        _frameCounter.PropertyChanged += (o,e) =>
            StatusText = String.Format("FPS = {0:N1} / CPU = {1:N6}",
                            _frameCounter.FramesPerSecond,
                            _frameCounter.CpuTimePerFrame);

        _sensor = KinectSensor.Default;
        _sensor.Open();
        _reader = sensor.DepthFrameSource.OpenReader();
        _reader.FrameArrived += FrameArrived;

        this.DataContext = this;
    }

    private void FrameArrived(object sender, DepthFrameArrivedEventArgs e)
    {
        var reference = e.FrameReference;
        DepthFrame depthFrame = null;

        try
        {
            // increment the frame counter
            using (_frameCounter.Increment())
            {
                depthFrame = reference.AcquireFrame();
                
                // do work with depth data...


            } // disposing the frame counter will calculate CPU time for this block
        }
        catch
        {
        }
        finally
        {
            if (depthFrame != null)
                depthFrame.Dispose();
        }
    }

}

Expand the block below for code goodness.

public class FrameCounter : INotifyPropertyChanged
{
    private Stopwatch _stopWatch;
    private uint _framesSinceUpdate = 0;
    private DateTime _nextStatusUpdate = DateTime.MinValue;
    private CpuCounter _cpuCounter;
    private double _fps;
    private double _cpuTime;

    public FrameCounter()
    {
        _stopWatch = new Stopwatch();
        _cpuCounter = new CpuCounter(this);
    }

    public double FramesPerSecond
    {
        get { return _fps; }
        protected set
        {
            if (_fps != value)
            {
                _fps = value;
                NotifyPropertyChanged("FramesPerSecond");
            }
        }
    }

    public double CpuTimePerFrame
    {
        get { return _cpuTime; }
        protected set
        {
            if (_cpuTime != value)
            {
                _cpuTime = value;
                NotifyPropertyChanged("CpuTimePerFrame");
            }
        }
    }

    public event PropertyChangedEventHandler PropertyChanged;

    public void DeferFrameCount(TimeSpan timeToWait)
    {
        this._nextStatusUpdate = DateTime.Now + timeToWait;
    }

    public IDisposable Increment()
    {
        this._framesSinceUpdate++;

        if (DateTime.Now >= this._nextStatusUpdate)
        {
            double fps = 0.0;
            double cpuTime = 0.0;

            if (this._stopWatch.IsRunning)
            {
                this._stopWatch.Stop();
                fps = this._framesSinceUpdate / this._stopWatch.Elapsed.TotalSeconds;
                cpuTime = this._cpuTime / fps;
                this._stopWatch.Reset();
            }

            this._nextStatusUpdate = DateTime.Now + TimeSpan.FromSeconds(1);
            this.FramesPerSecond = fps;
            this.CpuTimePerFrame = cpuTime;
        }

        if (!this._stopWatch.IsRunning)
        {
            this._framesSinceUpdate = 0;
            this._cpuTime = 0;
            this._stopWatch.Start();
        }

        _cpuCounter.Reset();
        return _cpuCounter;
    }

    internal void IncrementCpuTime(double amount)
    {
        _cpuTime += amount;
    }

    public void Reset()
    {
        _nextStatusUpdate = DateTime.MinValue;
        _framesSinceUpdate = 0;
        FramesPerSecond = 0;
        CpuTimePerFrame = 0;
    }

    protected void NotifyPropertyChanged(string propertyName)
    {
        if (PropertyChanged != null)
        {
            PropertyChanged(this, new PropertyChangedEventArgs(propertyName));
        }
    }

    public class CpuCounter : IDisposable
    {
        FrameCounter _parent;
        Stopwatch _stopWatch;

        internal CpuCounter(FrameCounter parent)
        {
            _parent = parent;
            _stopWatch = new Stopwatch();
        }

        public TimeSpan Elapsed
        {
            get
            {
                return _stopWatch.Elapsed;
            }
        }

        public void Reset()
        {
            _stopWatch.Reset();
            if (!_stopWatch.IsRunning)
            {
                _stopWatch.Start();
            }
            
        }

        public void Dispose()
        {
            _stopWatch.Stop();
            _parent.IncrementCpuTime(_stopWatch.Elapsed.TotalMilliseconds);
        }
    }
}

Happy coding.

submit to reddit

Tuesday, February 04, 2014

Comparing Kinect v1 and v2 Depth Data

In this post I show how the new Kinect v2 stacks up against the previous sensor. I’ve been blogging about the new Kinect as part of the Kinect for Windows Developer Preview. First and foremost, I have to get this legal disclaimer out of the way…

This is an early preview of the new Kinect for Windows, so the device, software and documentation are all preliminary and subject to change.

As part of this post, I thought it would be fun to build an application that targets both versions of the device and toggle back and forth between them for comparison sake. There’s a little bit of magic involved as both have the same namespaces and assembly name. Here’s you how you can do the same.

  1. Make a copy of the v1.8 Microsoft.Kinect.dll as Microsoft.Kinect.v1.dll
  2. In your project, add a reference this assembly and change the alias from global to v1.
  3. Do the same for v2.
  4. Add the following code above your using statements.
extern alias v1;
extern alias v2;

using KinectV1 = v1::Microsoft.Kinect;
using KinectV2 = v2::Microsoft.Kinect;

What’s changed between versions?

The first and perhaps most surprising difference is that the frame resolution of the depth data is 512 x 424. Don’t panic. Yes, the Kinect v1 supports a higher resolution of 640x480 resolution, but don’t get caught up on that. This apparently is the native resolution of the camera and although Microsoft may expose other resolutions in a later release, the differences between Time of Flight and Structured Light more than make up for this difference.

As I understand it, Structured Light (the depth technology in v1) reconstructs the depth data by analyzing deformations of a projected IR pattern. This produces approximations for pixels between the projected points. Time of Flight (used in Kinect v2) however blasts the scene with IR and measures how quickly the light comes back to the sensor. This produces better results as each pixel has a unique value. Time of Flight sensors also tend to have lower resolutions, so 512 x 424 is actually pretty impressive compared to other TOF sensors.

The second main change is the field of view and focal length for the camera. The Kinect can see more of the area and can detect objects much closer.

Side by Side Comparisons

The following table shows the observable differences in the depth frame data between versions.

  Kinect V1 Kinect V2
Height 480 424
Width 640 512
Aspect Ratio 4:3 6:5
Horizontal FOV 58.5 70.6
Vertical FOV 45.6 60
Diagonal FOV 70 89.5
Min Depth 0.8 m 0.5 m
Max Depth 4.0 m 4.5 m

The following snapshots are taken from two Kinect devices sitting side by side:

WP_20140203_001

Kinect V1

As part of the structured light algorithm, parts of the depth data are approximated. The subject must be at least 800mm from the camera.

KinectScreenshot-ColorMap-09-27-48

Kinect V2

The new Time of Flight sensor provides greater depth qualities per pixel and a much improved signal to noise ratio despite having an odd 6:5 aspect ratio (512 x 424 px). The increased field of view allows the camera to see more of the scene.

KinectScreenshot-ColorMap-09-28-02

submit to reddit

Monday, January 27, 2014

Changing the way .net looks for dependencies

Whenever I have to clean up my machine in preparation for changing my hardware or an OS refresh, I always find some forgotten gems in the process. Here’s one that I’m throwing up onto my blog for prosperity sake.

Every so often I have a project with lots of assemblies and dependencies which results in a deployment folder that can be daunting to the uninitiated. It would be great if the deployment folder had the exe and the config file and all the other dependencies were hidden in subfolders.

Here’s an example of how to do that.

  1. First, create two projects. Shell is an exe and Dependency is a ClassLibrary
  2. Change the Build folder of Shell to point to “..\AssemblyOutput”
  3. Change the Build folder of Dependency to point to “..\AssemblyOutput\modules”
  4. In the Shell project, add a reference to the Dependency project
  5. In the Shell’s references, set the CopyLocal property of the Dependency reference to “False”. If you don’t do this, the assembly will be copied to the root folder of the Shell’s output folder.
  6. Add the following to the Shell’s app config:
<configuration>

  <runtime>
    <assemblyBinding xmlns="urn:schemas-microsoft-com:asm.v1">
      
      <!-- Probing path is relative to the executable. Multiple values are supported (bin;bin2;modules;modules\subfolder)-->
      <probing privatePath="modules"/>

    </assemblyBinding>
  </runtime>

</configuration>

Now when you run the project, the first time the code comes across a reference to a type inside the Dependency assembly, it will load the assembly in the modules folder.

Happy coding.

submit to reddit

Tuesday, January 14, 2014

Leverage the Kinect v2 underlying depth buffer

One of the great new features in the API for new Kinect for Windows v2 developer preview is the ability to access the underlying buffer for the data sources. Normally, you can get by with using the CopyFrameDataToArray method but this option allows you to get a little closer to the metal and possibly squeeze out a bit more performance. Frankly, this is a good thing since we’re iterating over 217,088 pixels at a rate of 30 times per second – anything helps.

This is an early preview of the new Kinect for Windows, so the device, software and documentation are all preliminary and subject to change.

Also note, because C# does not support pointer arithmetic by default, you must mark your methods with the unsafe keyword.

To access the buffer, we use the AccessUnderlyingBuffer method on the DepthFrame which tells us how big the size of the buffer is and a pointer to the buffer.

unsafe
{
    uint size = 0;
    IntPtr ptr = IntPtr.Zero;

    depthFrame.AccessUnderlyingBuffer(out size, out ptr);

    // ptr now points to the buffer
}

Once we have a pointer to the buffer we can iterate through it’s contents. It’s up to us to understand how to parse the buffer and it’s not type-safe. We assume that the buffer is comprised of an array of ushort.

// this is also unsafe, so keep in the unsafe block above...
    
// obtain a pointer to the ushort array
ushort* data = (ushort*)ptr.ToPointer();

for(int i = 0; i < size; i++)
{
    ushort depth = data[i];

    // work with depth data
}

In my modest experimentation, switching to the buffer reduced cpu time between 0.5 – 1.0 milliseconds. Your mileage will obviously vary. Here’s a small clip of me toggling between the two approaches. The CPU counter in the bottom left is averaged over the number of frames per second and includes the time to render and update the image. There’s a subtle difference.

Happy Coding!

submit to reddit

Friday, January 10, 2014

Finding Pixels for Bodies in Kinect for Windows v2

In this post, I’m using Depth and BodyIndex data to create a depth visualization that highlights the active users.

This is an early preview of the new Kinect for Windows, so the device, software and documentation are all preliminary and subject to change.

KinectScreenshot-ColorMap-09-29-48

This post illustrates a few breaking changes from v1 of the Kinect for Windows API.

First off, the event signature to receive all data from the sensor has changed. Rather than getting data from the AllFramesReady event of the sensor, version 2 now uses a MultiSourceFrameReader that has a MultiSourceFrameArrived event. The MultiSourceFrameReader allows you to specify which data streams you wish to receive. This is a relatively minor change.

The other minor change this post illustrates is that the depth data no longer contains a byte to indicate the player index. Rather than applying a bitmask to the depth data, it is now exposed as its own data stream.

this.sensor = KinectSensor.Default;
this.sensor.Open();
this.reader = sensor.OpenMultiFrameSourceReader(
                        FrameSourceTypes.BodyIndex |
                        FrameSourceTypes.Depth
                        );
this.reader.MultiSourceFrameArrived += MultiSourceFrameArrived;

DepthFrameDescription desc = sensor.DepthFrameSource.FrameDescription;

this.bytesPerPixel = (PixelFormats.Bgr32.BitsPerPixel + 7) / 8;
this.depthFrameData = new ushort[desc.Width * desc.Height];
this.bodyIndexData = new byte[desc.Width * desc.Height];

this.pixels = new byte[desc.Width * desc.Height * this.bytesPerPixel];
this.bitmap = new WriteableBitMap(desc.Width, desc.Height, 96, 96, PixelFormat.Bgr32, null);

When the frame arrives, I fill both the depth and body index arrays with the available data. All that's left is to loop through the depth data and color the pixels. In this case, I'm using the GradientColorMap from my last post to decorate color pixels differently than background pixels.

void MultiSourceFrameArrived(object sender, MultiSourceFrameArrivedEventArgs e)
{
    MultiSourceFrameReference frameReference = e.FrameReference;

    MutliSourceFrame multiSourceFrame = null;
    DepthFrame depthFrame = null;
    BodyIndexFrame bodyIndexFrame = null;

    try
    {
        multiSourceFrame = frameReference.AcquireFrame();
        if (multiSourceFrame != null)
        {
            var depthFrameReference = multiSourceFrame.DepthFrameReference;
            var bodyIndexFrameReference = multiSourceFrame.BodyIndexFrameReference;

            depthFrame = depthFrameReference.AcquireFrame();
            bodyIndexFrame = bodyIndexFrameReference.AcquireFrame();

            if ((depthFrame != null) && (bodyIndexFrame != null))
            {
                depthFrame.CopyFrameDataToArray(this.depthFrameData);
                bodyIndexFrame.CopyFrameDataToArray(this.bodyIndexFrameData);

                int pixelIndex = 0;
                for (int i = 0; i < depthFrameData.Length; ++i)
                {
                    ushort depth = depthFrameData[i];
                    byte player = this.bodyIndexFrameData[i];
                    Color pixelColor;

                    if (player == 0xff) // nobody here
                    {
                        pixelColor = this.grayscaleMap.GetColorForDepth(depth);
                    }
                    else
                    {
                        pixelcolor = this.playerMap.GetColorForDepth(depth);
                    }
                }
                
                pixels[pixelIndex++] = pixelColor.B;
                pixels[pixelIndex++] = pixelColor.G;
                pixels[pixelIndex++] = pixelColor.R;
                pixelIndex++;
             }

             this.bitmap.WritePixels(
                 new Int32Rect(0, 0, depthWidth, depthHeight),
                 this.pixels,
                 depthWidth * this.cbytesPerPixel,
                 0);
            }

        }

    }
    catch
    {
    }
    finally
    {
        // MultiSourceFrame, DepthFrame, ColorFrame, BodyIndexFrame are IDispoable
        if (depthFrame != null)
        {
           depthFrame.Dispose();
           depthFrame = null;
        }

        if (bodyIndexFrame != null)
        {
            bodyIndexFrame.Dispose();
            bodyIndexFrame = null;
        }

        if (multiSourceFrame != null)
        {
            multiSourceFrame.Dispose();
            multiSourceFrame = null;
        }
    }

}

Easy peasy! Happy Coding!

Wednesday, January 08, 2014

Accessing Depth data with Kinect v2

This is an early preview of the new Kinect for Windows, so the device, software and documentation are all preliminary and subject to change.

Accessing Depth Data for the new Kinect for Windows API v2 is easy and very similar to the previous version of the API. We’ll do some more with this in a bit, but here’s a code snippet that’s been paraphrased from the v2 samples:

// Get the Sensor
this.sensor = KinectSensor.Default;

if (this.sensor! = null)
{
    // If we have a sensor, open it
    this.sensor.Open();

    // this example will use a BGR format image...    
    this.bitsPerPixel = (PixelFormats.Bgr32.BitsPerPixel + 7) / 8;

    // initialize some arrays based on the depth frame size
    FrameDescription desc = this.sensor.DepthFrameSource.FrameDescription;
    int size = desc.Width * desc.Height;
    this.frameData = new ushort[size];
    this.pixels = new byte[size * bitsPerPixel];

    this.bitmap = new WriteableBitmap(desc.Width, desc.Height, 96.0, 96.0, PixelFormats.Bgra32, null);
    
    // obtain a reader and subscribe to events
    this.reader = this.sensor.DepthFrameSource.OpenReader();
    this.reader.FrameArrived += FrameArrived;
}


void FrameArrived(object sender, DepthFrameArrivedEventArgs e)
{
    DepthFrameReference frameReference = e.FrameReference;

    try
    {
        DepthFrame frame = frameReference.AcquireFrame();

        if (frame != null)
        {
            // frame is disposable
            using(frame)
            {

                var desc = frame.FrameDescription;

                frame.CopyFrameDataToArray(this.frameData);

                int colorPixelIndex = 0;

                // loop through the frame data
                for(int i = 0; this.frameData.Length; ++i)
                {
                    ushort depth = this.frameData[i];

                    // do something interesting with depth data here
                    // see examples below...
                    // Color color = this.colorMap.GetColorForDepth(depth);
                    Color color = new Color();

                    // assign values to individual pixels
                    // BGRA format
                    this.pixels[colorPixelIndex++] = color.B; 
                    this.pixels[colorPixelIndex++] = color.G;
                    this.pixels[colorPixelIndex++] = color.R;
                    ++colorPixelIndex; // skip Alpha color
                }

                this.bitmap.WritePixels(
                    new Int32Rect(0, 0, desc.Width, desc.Height),
                    this.pixels,
                    desc.Width * this.bitsPerPixel,
                    0);

            }
        }
    }
    catch(Exception)
    {
        // frame might no longer be available
    }
}

Fairly straight-forward, right? The biggest difference from v1 is that we must access the data from an event on the reader rather than an event from the sensor.

Intensity

The example in the SDK truncates the depth data down to the last byte, which produces a grayscale effect that “wraps” every 255 millimeters. It doesn’t look very nice, but the key thing to observe is just how detailed the depth data is. If you look closely, you can clearly make out facial features and other fine details such as the lines in my hands.

KinectScreenshot-ColorMap-08-37-27

// paraphrased from the SDK examples..
// truncates the depth data down to the lowest byte, which
// produces an "intensity" value between 0-255

ushort minDepth = desc.DepthMinReliableDistance;
ushort maxDepth = desc.DepthMaxReliableDistance;

// To convert to a byte, we're discarding the most-significant
// rather than the least-significant bits.
// We're preserving detail, although the intensity will "wrap."
// Values outside the reliable depth range are mapped to 0 (black).
byte intensity = (byte)(depth >= minDepth && depth <= maxDepth) ? depth : 0;

// convert intensity to a color (to fit into our example above)
var color = new Color()
        {
            R = intensity,
            G = intensity,
            B = intensity
        };

Prettier examples

Here’s a few examples I’ve cooked up. From past experience with the Kinect v1 we observed that you can improve performance by reducing the amount of calculations per frame. One approach I’ve used it to pre-calculate the results into a hash table between 500 and 4500 milimeters. There are likely other optimizations that can be made, but this is easy.

For fun, I’ve created an abstraction called a DepthMap:

abstract class DepthMap
{
    protected ushort minDepth;
    protected ushort maxDepth;

    protected DepthMap(ushort minDepth, ushort maxDepth)
    {
        this.minDepth = minDepth;
        this.maxDepth = maxDepth;
    }

    public abstract Color GetColorForDepth(ushort depth);
}

HSV Color Wheel

Rather than a grayscale effect, I found a few great online samples on how to implement a HSV color wheel which produces this fun rainbow effect. I’m gradually transitioning between colors by looking at the most significant bits.

KinectScreenshot-ColorMap-08-47-20

double h, s, v;
int r, g, b;

h = ((depth - minDepth) / (maxDepth - minDepth)) * 360;
v = 1;
s = 1;

Hsv.HsvToRg(h, v, s, out r, out g, out b);

return new Color {
        R = (byte)r,
        G = (byte)g,
        B = (byte)b
    };

Color Gradients

The formula to produce a gradient is rather simple, you take a percentage of one color and a remaining percentage of another color. This color map allows me to pick the colors I want to use, which produces some interesting effects. Using an array of colors (Red, Yellow, Green, Yellow, Red) I can color code depth to reflect an ideal position from the camera.

KinectScreenshot-ColorMap-08-47-39

var colorMap = new GradientColorMap(
    minDepth, 
    maxDepth, 
    new[] { 
        Colors.Red, 
        Colors.Yellow, 
        Colors.Green, 
        Colors.Yellow, 
        Colors.Red 
        }
    );

I can also create a boring grayscale effect by just using Black and White.

var colorMap = new GradientColorMap(
    minDepth,
    maxDepth,
    new[] {
        Color.White,
        Color.Black
        }
    );

You can see some of them in action here:

Happy coding!

Monday, January 06, 2014

Multiple Apps can use Kinect for Windows v2

One of the new features of the Kinect for Windows API is that multiple applications can simultaneously access the sensor, as demonstrated by this video:

This is an early preview of the new Kinect for Windows, so the device, software and documentation are all preliminary and subject to change.

The lower right hand corner of this video shows a console application, KinectService, that must be running in order to run the samples provided. It’s unclear what role the service will play in the future releases of the SDK but I’d expect that it runs as Windows Service or headless process running from the system tray.

The Kinect API has changed to reflect this ability. It’s a minor change that will break some of your existing v1 code, but should be relatively easy to port to v2. Previous versions of the API allowed you to access the sensor data by subscribing to events on the sensor. The new version of the API introduces a reader pattern that separates the events from the sensor. Each reader has access to their own frames, which should in theory allow for more modular applications.

Here’s a quick sample:

// v1
KinectSensor sensor = KinectSensor.Sensors[0];
sensor.DepthStream.Enable(DepthImageFormat.Resolution640x480Fps30);
sensor.Start();
sensor.DepthFrameReady += OnDepthFrameReady;

// v2
KinectSensor sensor = KinectSensor.Default;
sensor.Open();
DepthFrameReader reader = sensor.DepthFrameSource.OpenReader();
reader.FrameArrived += OnDepthFrameArrived;

This change is significant because most devices are exclusively locked to the application that is using it. This can be a real pain when your app crashes but doesn’t release the handle of the device.

I can see many options for this functionality:

  • A support engineer can remotely troubleshoot the environment setup without having to close the primary application
  • A background process could take periodic snapshots to produce timelapse videos of usage of your application
  • Applications produced by different vendors could simulatenously run on the same machine: A desktop application could dim the monitor when you're not looking at it while a gesture application augments your windows experience...

What will you do with this feature?

Thursday, December 26, 2013

A Quick Peek at the New Kinect for Windows

So I’ve had a few days to play with the new Kinect for Windows device and SDK, and I have lots to say, but here’s a quick post to share some immediate observations about the hardware. I’ll post some more, hopefully with video and code examples when I get a break between all the holiday festivities.

First off, I need to get some disclaimers out of the way. This is an early preview of the new Kinect for Windows, so the device, software and documentation are all preliminary and subject to change.

That being said so far the device is pretty rock solid. Although I haven’t got access to an Xbox One for comparison my guess is this is very similar to the unit that ships with the console but with a different connection and power supply. The device I have has stickers on it and although the picture below doesn’t highlight this well, but if you look closely at the sticker on the right-hand side of the device you’ll see the Xbox logo shining through slightly. No doubt, we will see some change in the appearance of the device when the official device is shipped next year.

WP_20131226_001

An interesting departure from KFW 1.0 is that it appears that Microsoft has ditched the motorized tilt motor in favor of an manually adjustable base. I imagine that the motor was a support problem as it could break if you applied too much force. I will miss the ability to freak people out by making it move randomly, but from a developer perspective this puts an onus on the hardware setup as placement of the device cannot be adjusted remotely. Maybe we can find an algorithm to infer camera angle?

Another key variation is that the base of the unit has a proper camera mount screw allowing you to deploy the device with one of many existing industry standard tripods or specialized camera mounts. I will not miss the proprietary Kinect mount.

WP_20131226_002

The power supply and connection to the PC is interesting and also likely to change in the official device. The cord coming out of the camera is quite long and has a specialized connector (likely the same as the xbox one). In order to connect this specialized connector to a PC, the camera plugs into a box with three connections: camera, USB and power. The power cord is about 3 feet long and connects a fair sized power brick which then has another 3 feet before connecting into an outlet.

WP_20131226_003

In my next post we’ll take a look at some of the new features of the SDK.

Saturday, December 21, 2013

Guess what I’ll doing during Christmas break?

It arrived at work just before my Christmas vacation…

WP_20131219_003

It came with stickers…

WP_20131219_005

Expect a few more posts from me over the Christmas break about the new Kinect for Windows.

By the way, there’s still time for you to get in on this as Microsoft has extended the Developer Preview for another 500 devices.

Tuesday, November 05, 2013

The first User Story for your next Project

For your next project, consider adding this story to the very first sprint:

As an administrator I want to install the program with an installer so that I can have everything in place without manual setup.

Acceptance criteria:

  • Application should run and show an empty screen
  • Application should produce a log file that shows debug diagnostic information that can be used to trace through application runtime (timestamp, thread, logging level, component, message). Any fatal errors during start-up should be recorded with helpful diagnostic details.
  • Any prerequisites for the installer must be known in advance or bundled with the installer
  • The installer should have a minimal user-interface and optionally support running unattended
  • The installer should put a shortcut on the desktop and start menu

Sure, this is great, but why should this be the first story in my project?

The short answer is this is the best time to do this. An installer created at the beginning of the project is often very simple as it will likely only be a handful of files. If you’ve ever written an installer, you’ll know that it can be a lot of work establishing all your dependencies and user-interface. If you wait until the end of the project when time is often a critical resource, you’ll fall into the trap of dropping this feature.

The beginning of your project is also a really good time to start thinking about deployment. By creating an installer you’ll have a packaging mechanism for your testers and business stakeholders and every time they install your application they will be regression testing the deployment. Bugs will be found, and the installer will get better and more robust as the project advances. From a development perspective you might think of this as re-work. Try to break out of that mindset and think of it more like spreading the work over the project in small increments.

As a final note, you may notice that some of the requirements refer to non-functional requirements like diagnostic logging. I’ve found this is a great way to set scope for the developers when referring to the initial “empty screen” and has an added benefit that lets testers know the location of the log files. Also, it allows you to associate defects for weird start-up errors and crashes to a User Story.

Happy coding.

Wednesday, October 16, 2013

Advance WorkItems to next state on check-in

Nearly two years ago, I had a project where we used whiteboard and post-it notes for our Kanban board. Perhaps one of my favourite aspects of using a whiteboard was the non-verbal communication that occurred between team members: when a developer would finish a task, she would stand-up from her desk, take a deep breath and well deserved stretch then rip her post-it note from the In-Progress column, slap it into the Ready-for-test column and then yank one from the Next column. Everyone on the team would look up, smile and go back to coding.

Alas, whiteboards and post-it notes only work when all team members can see the board and when you’re teams are remote you will need a software solution. Our organization is big on TFS, and I’ve found much luck using SEP Teamworks to simplify the data and present it in a Kanban fashion.

One of the challenges with using software for task tracking is that it loses that tacit capability. Choosing the wrong tool can mean you spend more time managing the tool than building software.

Here’s a quick post that illustrates how you can leverage features of TFS workflow to automate your Kanban process a bit, so you don’t have to harass your team members so much.

The xml schema for our User Stories, Bugs and Tasks contains elements that describe fields, user-interface and workflow. The workflow element is interesting because it allows us to define the supported transitions between states and default values for fields in each state. It also supports this sweet little addition that will automatically transition a work-item to a different state simply by including the following Action in the Actions element:

<TRANSITION from="In Development" to="Ready for Test">
  <REASONS>
    <DEFAULTREASON value="Development Complete" />
  </REASONS>
  <FIELDS>
    <FIELD refname="System.AssignedTo">
      <ALLOWEXISTINGVALUE />
      <EMPTY />
    </FIELD>
  </FIELDS>
  <ACTIONS>
    <ACTION value="Microsoft.VSTS.Actions.Checkin" />
  </ACTIONS>
</TRANSITION>

Although the schema suggests that it would allow custom actions to be plugged in here, only the Microsoft.VSTS.Actions.Checkin action is supported.

To take advantage of this feature, simply associate your work items to your check-in and mark the action as “Resolve”.

TFS_AssociateWorkItems

Thursday, September 12, 2013

Kanban, TFS style.

I’ve been building up a system of tracking tasks using User Stores in TFS for almost two years and I’ve been blogging as I go. Over time, I’ve written several short emails to colleagues pointing them to the Kanban tag of my blog, but that tag list is a bit heavy. Here’s a quick recap of helpful posts that can get you up and running.

Getting Started

If you’re a developer, team lead or architect – you should start here.

  1. My first month with Kanban – a quick walkthrough that discusses my initial observations with Kanban. Old school, using a whiteboard and sticky-notes.
  2. Using Kanban with TFS – a high level overview of the process I’ve adopted for tracking Tasks in TFS using Kanban columns. Provides an overview of customizing work-items.
  3. Configuring SEP Teamworks – a walk through how to set up SEP Teamworks, my favorite free tool for viewing TFS items in a sticky-note fashion.
  4. Using User Stories with SEP Teamworks – Revisiting my workflow slightly to adapt to using User Stories instead of just individual task tracking.
  5. Setting up Areas and Iterations – Shows you how to organize your stories into TFS Areas & Iterations, and then using SEP Teamworks features to simpli

Reporting and Querying

If you’re a project manager and you want to set up SEP TeamWorks. Please read:

  1. Working with TFS WorkItems in Excel – includes a walk through on how to set up your machine, and export queries to Excel.
  2. Configuring SEP Teamworks – a walk through how to set up SEP Teamworks, my favorite free tool for viewing TFS items in a sticky-note fashion.

Advanced Customizations

  1. What’s new in SEP Teamworks 1.0.31 – highlights some of the new improvements in the latest release: Card Editor to customize views, creating linked work items.
  2. Marking Kanban items as Blocked – demonstrates how to customize your work items to take advantage of marking items as blocked.

Tuesday, August 13, 2013

Fix your code with an “On Notice” board

OnNotice

The above comes with thanks to the On Notice Generator, and my board re-iterates a lot of the guidance found on MiÅ¡ko Hevery’s blog. These are code patterns and anti-patterns that I’ve encountered on many projects and have strong feelings against. Some of these are actually on my Dead to Me board, but there wasn’t a online generator.

I think it’s a good habit to start on On Notice board for your project – a list of offending code that should be cleaned-up at some point. Often, these unsightly offenders are large and tightly woven into the fabric of our code so they’re not something that can be fixed in a single refactoring. But by placing these offences on a visible On Notice board, they become goals that can fuel future refactorings.

You might not be able to fix a problem in a single session, but you can add a 2 hour research task to your backlog to understand why it exists. The output of such task might be further research tasks or changes you could introduce to shrink their influence and eventually remove all together. Sometimes I bundle a bunch of these fixing tasks into a refactoring user story, or slip a few into a new feature if they’re related. Over time, the board clears up.

Don’t forget, while you’re making these changes, write a few tests while you’re at it.

Oh, regarding Gluten-free cookies… they look like cookies, but they are most definitely not.

Monday, July 22, 2013

Thanks for Saying Thanks!

A few months back, I received a nice email from the folks at SEP for my blog notes on SEP TeamWorks. It’s nice to know that someone is paying attention:

My name is Adam Scroggin and I am the product owner for SEP TeamWorks.  I came across your blog today (http://www.bryancook.net/) and just wanted to say thanks for the write ups you did.  If there are any features you want added, feel free to contact me.

Keep on writing! Adam

Adam Scroggin | Software Engineering Manager

Appreciate the feedback, Adam. I have a few more blog posts to add regarding using the tool with TFS. I’m sure you’d be happy to learn that teams within my organization continue to adopt the tool for its ease of use.

Thursday, July 18, 2013

Unhandled exceptions in WPF applications

When it comes to unit testing there are a few areas of the application where I am comfortable not getting coverage. There are some areas of the application, typically in the UI, that are generally difficult to unit test but can be easily verified if you run the application manually. There are a few other places where testing is very difficult to validate such as the global error handler for your application. For the global error handler, you have to live with some manual testing and assume you’ve got it right.

Today I discovered one of my assumptions about the global error handler was completely wrong. My app was crashing and displaying error messages; I assumed it was crashing, logging to a file and exiting politely. It was not. And as always, I’m writing this as a reminder for you and myself.

As most know, the best place for a global exception handler is to attach an event handler to the DispatcherUnhandledException event of the application. It’s important to set the the Handled property of the UnhandledExceptionEventArgs to true to prevent the app from crashing.

However, this will only capture exceptions on the UI thread. All other exceptions will look for an event handler on that threads’ stack. If no event handler is found it will bubble up to the AppDomain’s handler. So to capture these exceptions you should add an event handler to the AppDomain’s UnhandledException event.

In contrast to the UnhandledExceptionEventArgs, UnahdledException does not have a Handled property. I assumed that the purpose of this handler was so that we could log the error and go on about our business. As it turns out, if your code reaches to this event handler it is completely unrecoverable. As in Bill Paxton, “Game over, man!” – your app is going to crash and show a nasty error dialog. The only way to prevent the dialog is to use Environment.Exit(1);

namespace MyApplication
{
    public class MyApp : Application
    {
        private static ILog Log = log4net.LogManager.GetLogger(typeof(MyApp));

        protected override void OnStartup(StartupEventArgs e)
        {
            base.OnStartup(e);

            // handle all main UI thread related exceptions
            Application.Current.DispatcherUnhandledException += DispatcherUnhandledException;

            // handle all other exceptions in background threads
            AppDomain.CurrentDomain.UnhandledException += AppDomainUnhandledException;
        }

        void DispatcherUnhandledException(object sender, System.Windows.Threading.DispatcherUnhandledExceptionEventArgs e)
        {
            // prevent unhandled exception from crashing the application.
            e.Handled = true;

            Log.Fatal("An unhandled exception has reached the UI Dispatcher.", e.Exception);

            // shut down the application nicely.
            Application.Shutdown(-1);
        }

        void AppDomainUnhandledException(object sender, UnhandledExceptionEventArgs e)
        {
            var ex = e.ExceptionObject as Exception;

            Log.Fatal("An unhandled exception has reached the AppDomain exception handler. Application will now exit.", ex);

            // This exception cannot be handled and you cannot reliably use Shutdown to gracefully shutdown.
            // The only way to suppress the CLR error dialog is to supply "1" to the exit code.
            Environment.Exit(1);
        }
    }    
}

If you want to gracefully exit the application regardless which thread created the Exception, the recommended approach is to:

  • Handle the exception on the background thread.
  • Marshal the exception to the UI thread and then re-throw it there.
  • Handle the exception in the Application.DispatcherUnhandledException handler.

There’s no easy way out here and means you need to fix the offending code. My recommendation is to use the AppDomain UnhandledException as a honey pot to find issues.

High five.

submit to reddit

Monday, July 08, 2013

DeploymentItems in Visual Studio 2012

A frequent concern with writing unit tests with MSTest is how to include additional files and test data for a test run. This process has changed between Visual Studio 2010 and 2012 and it’s become a source of confusion.

Background

With Visual Studio 2010 and earlier, every time you ran your tests Visual Studio would copy all files related to the test to a test run folder and execute them from this location. For local development this feature allows you to compare results between test runs, but the feature is also intended to support deploying the tests to remote machines for execution.

If your tests depend on additional files such as external configuration files or 3rd party dependencies that aren’t directly referenced by the tests, you would need to enable Deployment in your testsettings and then either specifying the deployment items in the testsettings file or marking each test with a DeploymentItemAttribute.

What’s changed in Visual Studio 2012?

Visual Studio 2012 has a number of changes related to the test engine that impact deployment. The most visible change is that Visual Studio 2012 no longer automatically adds the testsettings file to your solution when you add a Test project. The testsettings file can be added to your project manually, but it’s generally recommended that you don’t use it as it’s for backward compatibility and not all features within Visual Studio 2012 are backward compatible. Microsoft Fakes for example are not backward compatible.

The biggest change related to deployment is that Visual Studio 2012 tests run directly out of the output folder by default. This adds a significant speed boost for the tests but it also means that if your tests are dependent on files that are already part of the build output, you won’t need to enable deployment at all.

Another interesting change is that if you include a DeploymentItemAttribute in your tests, Deployment will be automatically enabled and your tests will run out of the deployment folder.

More information can be found here.

Saturday, May 25, 2013

.props files and NuGet 2.5

My last post showed a very simple PowerShell script to automate project properties as part of a NuGet package. Shortly after posting, I exchanged a few tweets with some very smart people. The suggestion was that there’s a new feature in NuGet 2.5 that can pull .props and .targets files directly into your project without having to resort to powershell scripting. I had to try this.

So, I created a .props file:

<Project ToolsVersion="4.0" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
    <PropertyGroup>
        <AssemblyOriginatorKeyFile>mykey.snk</AssemblyOriginatorKeyFile>
        <SignAssembly>true</SignAssembly>
    </PropertyGroup>
</Project>

And put it in a new package:

image

Then added my shiny new package to a new project.

…And nothing happened.

…At first. I had to unload and reload the project to see the changes.

<Project ToolsVersion="4.0" DefaultTargets="Build" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
  <Import Project="..\packages\StrongKey2.1.0.0\build\StrongKey2.props" Condition="Exists('..\packages\StrongKey2.1.0.0\build\StrongKey2.props')" />
  <PropertyGroup>
    <Configuration Condition=" '$(Configuration)' == '' ">Debug</Configuration>
    <Platform Condition=" '$(Platform)' == '' ">x86</Platform>

I’m thinking, for my example anyway, that powershell is better suited for this?

Thursday, May 23, 2013

Manipulating Projects with NuGet Powershell scripts

I’ve been playing with hosting NuGet packages on our internal NuGet server a fair bit recently and encountered some interesting feedback from a colleague: none of our packages are signed with a strong-name. Gasp! Strong naming is definitely something that falls under the you really should bucket but gets quickly put in the not today simply because it’s a pain. If you’re familiar with the concept, the challenge with strong-naming is that if you give one assembly a strong-name then all dependent assemblies must also have one. This quickly cascades into a lot of repetitive tasks.

The pain of repetitive tasks is something that NuGet handles really well, so why not put my signing-key in a nuget package to automate this process? The concept of this package is extremely straight forward:

  1. Add my snk file to the project as content
  2. Manipulate the project properties in the install.ps1 script

I’ve really been wanting to write some PowerShell scripts for NuGet packages for a while now, but haven’t had the opportunity. My initial thought was to manipulate the project as an Xml document, but abandoned that approach as some research showed that Visual Studio would prompt the user to reload the project.

After some Googling, I came across a HaaHa (Haack/Hanselman) presentation at MIX11, where Scott’s AddMVC3ToWebForms package had some dirty hacks to manipulate the project. There’s a really good discussion in the NuGet forums that suggests using MSBuild API is a good approach to manipulate the Project properties only if what you want isn’t directly available from the Visual Studio API

Fortunately for me, manipulating the properties I wanted was dead easy so my script couldn’t be simpler:

param($installPath, $toolsPath, $package, $project)

$project.Properties.Item("AssemblyOriginatorKeyFile").Value = "mykey.snk";
$project.Properties.Item("SignAssembly").Value = "true";

$project.Save();

Few handy tips I discovered while writing this…

You can get access to the $project variable using the Package Manager Console:

$project = Get-Project

You can list the details of your objects using the get-member cmdlet:

$project | get-member

And you can get a dump of the current values, and list of properties easily:

$project.Properties
$project.Properties | select Name

Well, that’s all for now. Let’s hope this is the start of some awesome PowerShell NuGet badassness for you.

Happy Coding'.

Thursday, May 16, 2013

Working with TFS work items in Excel

Visual Studio ships with a pretty good query engine that you can use to retrieve a list of work items, but if you need to change the filter criteria frequently or calculate totals for estimates or remaining work, Visual Studio can’t cut it on its own. Fortunately, the integration between TFS and the Microsoft Office suite is fantastic and we can export our data into Excel with a few simple clicks.

Hey, if you’re reading this because you’re a project manager at my office and I sent you this link, you’ll need a few things on your machine for this to work:

  1. Download a copy of Visual Studio Team Explorer.  It’s basically a slimmed down shell of Visual Studio without all the code editing features and it includes all the goodies for you to query work items and export to Excel.  If you have Visual Studio installed, you can skip this step.
  2. You’ll need read permissions to the TFS Team Project. Reach out to your friendly IT support for this.
  3. If you’ve never connected to the TFS server before, you’ll also need the TFS Server name, Team Project Collection name, and Team Project name.

There are two ways to get TFS work items into Excel:

  • Import TFS Work Items into Excel
  • Export TFS Work Items from Visual Studio or Team Explorer into Excel

Importing TFS Work Items into Excel

Let’s assume that you’ve never connected to TFS before. Managing this from Excel is actually pretty easy.

  1. Open Excel and create a blank workbook
  2. Select the “Team” option from the Ribbon. If “Team” isn’t available in the Ribbon, you haven’t installed Team Explorer. (see above)
  3. Select the “New List”.
    Excel_TeamExplorer_Addin
  4. The Connect to Team Foundation Server dialog will appear.
    TFS_ConnectToServer
  5. Click the Servers button. The Add/Remove Team Foundation Server dialog will appear.
    TFS_AddRemoveServer
  6. Click the Add button to open the Add Team Foundation Server dialog.
    TFS_AddServerpng
  7. Fill in the server name and details provided to you by IT. You’ll know you’ve entered the right information when the Preview matches the information provided.
    TFS_AddServerpng
  8. After all the connection information is provided, select the appropriate Team Project Collection, and the Team Project that you want to use.
    TFS_SelectProject
  9. Finally a dialog appears that lets you select an existing TFS query.
    Excel_SelectQuery
  10. Once you’ve selected, the query, click OK. Voila!
    Excel_TFS_Goodness

Exporting TFS work items to Excel from Visual Studio

Exporting your favorite query from Visual Studio to Excel couldn’t be easier. Simply run the query and then select “Open in Microsoft Office –> Open Query in Microsoft Excel”

VS_ExportToExcel

Famous Last Words

Here’s a few tips with working with work items Excel:

  • At any time you can get the latest by clicking the “Refresh” button
  • If you make changes, you can push them back to the server by clicking the “Publish” button. (You’ll need write permissions for this)
  • This worksheet is tied directly to the server, so don’t forward the spreadsheet to individuals that don’t have access to the data. For this, copy the entire worksheet to another worksheet.