Programming with the Kinect for Windows Software Development Kit: Displaying Kinect Data

The skeleton display manager

The skeleton data is produced by the natural user interface (NUI) API and behaves the same way as the color and depth streams. You have to collect the tracked skeletons to display each of their joints.

You can simply add a WPF canvas to display the final result in your application, as shown in Figure 3-5:

<Canvas x:Name="skeletonCanvas"></Canvas>

You have to write a class named SkeletonDisplayManager that will provide a Draw method to create the required shapes inside the skeletonCanvas canvas:

using System;
using System.Collections.Generic;
using System.Windows;
using System.Windows.Controls;
using System.Linq;
using System.Windows.Shapes;
using System.Windows.Media;
using Microsoft.Kinect;

namespace Kinect.Toolbox
{
    public class SkeletonDisplayManager
    {
        readonly Canvas rootCanvas;
        readonly KinectSensor sensor;

        public SkeletonDisplayManager(KinectSensor kinectSensor, Canvas root)
        {
            rootCanvas = root;
            sensor = kinectSensor;
        }

        public void Draw(Skeleton[] skeletons)
        {
             // Implementation will be shown afterwards
        }
    }
}

As you can see, the Draw method takes a Skeletons array in parameter. To get this array, you can add a new method to your Tools class:

public static void GetSkeletons(SkeletonFrame frame, ref Skeleton[] skeletons)
{
    if (frame == null)
        return;

    if (skeletons == null || skeletons.Length != frame.SkeletonArrayLength)
    {
        skeletons = new Skeleton[frame.SkeletonArrayLength];
    }
    frame.CopySkeletonDataTo(skeletons);
}

This method is similar to the previous one but does not recreate a new array every time, which is important for the sake of performance. When this method is ready, you can add the following code to your load event:

Skeleton[] skeletons;
SkeletonDisplayManager skeletonManager = new SkeletonDisplayManager(kinectSensor, skeletonCanvas);
void kinectSensor_SkeletonFrameReady(object sender, SkeletonFrameReadyEventArgs e)
{
    using (SkeletonFrame frame = e.OpenSkeletonFrame())
    {
        if (frame == null)
            return;

        frame.GetSkeletons(ref skeletons);
        if (skeletons.All(s => s.TrackingState == SkeletonTrackingState.NotTracked))
            return;

        skeletonManager.Draw(skeletons);
    }
}

The event argument e gives you a method called OpenSkeletonFrame that returns a SkeletonFrame object. This object is used to get an array of Skeleton objects.

Then you simply have to find out if one of the returned skeletons is tracked. If not, you can return and wait for a new frame, or you can use the skeletonManager object to display the detected skeletons.

So, going back to your SkeletonDisplayManager, you now need to draw the skeletons inside the WPF canvas. To do so, you can add a list of circles that indicate where the joints are and then draw lines between the joints.

You can get access to a skeleton’s joints collection easily using the skeleton.Joints property. To draw all the detected and tracked skeletons in a frame, you simply cycle through the Skeletons array with the following code:

public void Draw(Skeleton[] skeletons)
{
rootCanvas.Children.Clear();
foreach (Skeleton skeleton in skeletons)
{
    if (skeleton.TrackingState != SkeletonTrackingState.Tracked)
        continue;

    Plot(JointType.HandLeft, skeleton.Joints);
    Trace(JointType.HandLeft, JointType.WristLeft, skeleton.Joints);
    Plot(JointType.WristLeft, skeleton.Joints);
    Trace(JointType.WristLeft, JointType.ElbowLeft, skeleton.Joints);
    Plot(JointType.ElbowLeft, skeleton.Joints);
    Trace(JointType.ElbowLeft, JointType.ShoulderLeft, skeleton.Joints);
    Plot(JointType.ShoulderLeft, skeleton.Joints);
    Trace(JointType.ShoulderLeft, JointType.ShoulderCenter, skeleton.Joints);
    Plot(JointType.ShoulderCenter, skeleton.Joints);

    Trace(JointType.ShoulderCenter, JointType.Head, skeleton.Joints);

    Plot(JointType.Head, JointType.ShoulderCenter, skeleton.Joints);

    Trace(JointType.ShoulderCenter, JointType.ShoulderRight, skeleton.Joints);
    Plot(JointType.ShoulderRight, skeleton.Joints);
    Trace(JointType.ShoulderRight, JointType.ElbowRight, skeleton.Joints);
    Plot(JointType.ElbowRight, skeleton.Joints);
    Trace(JointType.ElbowRight, JointType.WristRight, skeleton.Joints);
    Plot(JointType.WristRight, skeleton.Joints);
    Trace(JointType.WristRight, JointType.HandRight, skeleton.Joints);
    Plot(JointType.HandRight, skeleton.Joints);

    Trace(JointType.ShoulderCenter, JointType.Spine, skeleton.Joints);
    Plot(JointType.Spine, skeleton.Joints);
    Trace(JointType.Spine, JointType.HipCenter, skeleton.Joints);
    Plot(JointType.HipCenter, skeleton.Joints);

    Trace(JointType.HipCenter, JointType.HipLeft, skeleton.Joints);
    Plot(JointType.HipLeft, skeleton.Joints);
    Trace(JointType.HipLeft, JointType.KneeLeft, skeleton.Joints);
    Plot(JointType.KneeLeft, skeleton.Joints);
    Trace(JointType.KneeLeft, JointType.AnkleLeft, skeleton.Joints);
    Plot(JointType.AnkleLeft, skeleton.Joints);
    Trace(JointType.AnkleLeft, JointType.FootLeft, skeleton.Joints);
    Plot(JointType.FootLeft, skeleton.Joints);

    Trace(JointType.HipCenter, JointType.HipRight, skeleton.Joints);
    Plot(JointType.HipRight, skeleton.Joints);
    Trace(JointType.HipRight, JointType.KneeRight, skeleton.Joints);
    Plot(JointType.KneeRight, skeleton.Joints);
    Trace(JointType.KneeRight, JointType.AnkleRight, skeleton.Joints);
    Plot(JointType.AnkleRight, skeleton.Joints);
    Trace(JointType.AnkleRight, JointType.FootRight, skeleton.Joints);
    Plot(JointType.FootRight, skeleton.Joints);
}
}

The Trace and Plot methods search for a given joint through the Joints collection. The Trace method traces a line between two joints and then the Plot method draws a point where the joint belongs.

Before looking at these methods, you must add some more code to your project. First add a Vector2 class that represents a two-dimensional (2D) coordinate (x, y) with associated simple operators (+, -, *, etc.):

using System;

namespace Kinect.Toolbox
{
    [Serializable]
    public struct Vector2
    {
        public float X;
        public float Y;

        public static Vector2 Zero
        {
            get
            {
                return new Vector2(0, 0);
            }
        }

        public Vector2(float x, float y)
        {
            X = x;
            Y = y;
        }

        public float Length
        {
            get
            {
                return (float)Math.Sqrt(X * X + Y * Y);
            }
        }

        public static Vector2 operator -(Vector2 left, Vector2 right)
        {
            return new Vector2(left.X - right.X, left.Y - right.Y);
        }

        public static Vector2 operator +(Vector2 left, Vector2 right)
        {
            return new Vector2(left.X + right.X, left.Y + right.Y);
        }

        public static Vector2 operator *(Vector2 left, float value)
        {
            return new Vector2(left.X * value, left.Y * value);
        }

        public static Vector2 operator *(float value, Vector2 left)
        {
            return left * value;
        }

        public static Vector2 operator /(Vector2 left, float value)
        {
            return new Vector2(left.X / value, left.Y / value);
        }

    }

}

There is nothing special to note in the previous code; it is simple 2D math.

The second step involves converting the joint coordinates from skeleton space (x, y, z in meter units) to screen space (in pixel units). To do so, you can add a Convert method to your Tools class:

  public static Vector2 Convert(KinectSensor sensor, SkeletonPoint position)
        {
            float width = 0;
            float height = 0;
            float x = 0;
            float y = 0;

            if (sensor.ColorStream.IsEnabled)
            {
                var colorPoint = sensor.MapSkeletonPointToColor(position,
sensor.ColorStream.Format);
                x = colorPoint.X;
                y = colorPoint.Y;

                switch (sensor.ColorStream.Format)
                {
                    case ColorImageFormat.RawYuvResolution640x480Fps15:
                    case ColorImageFormat.RgbResolution640x480Fps30:
                    case ColorImageFormat.YuvResolution640x480Fps15:
                        width = 640;
                        height = 480;
                        break;
                    case ColorImageFormat.RgbResolution1280x960Fps12:
                        width = 1280;
                        height = 960;
                        break;
                }
            }
            else if (sensor.DepthStream.IsEnabled)
            {
                var depthPoint = sensor.MapSkeletonPointToDepth(position,
sensor.DepthStream.Format);
                x = depthPoint.X;
                y = depthPoint.Y;

                switch (sensor.DepthStream.Format)
                {
                    case DepthImageFormat.Resolution80x60Fps30:
                        width = 80;
                        height = 60;
                        break;
                    case DepthImageFormat.Resolution320x240Fps30:
                        width = 320;
                        height = 240;
                        break;
                    case DepthImageFormat.Resolution640x480Fps30:
                        width = 640;
                        height = 480;
                        break;
                }
            }
            else
            {
                width = 1;
                height = 1;
            }

            return new Vector2(x / width, y / height);
        }

The Convert method uses the Kinect for Windows SDK mapping API to convert from skeleton space to color or depth space. If the color stream is enabled, it will be used to map the coordinates using the kinectSensor.MapSkeletonPointToColor method, and using the color stream format, you can get the width and the height of the color space. If the color stream is disabled, the method uses the depth stream in the same way.

The method gets a coordinate (x, y) and a space size (width, height). Using this information, it returns a new Vector2 class with an absolute coordinate (a coordinate relative to a unary space).

Then you have to add a private method used to determine the coordinates of a joint inside the canvas to your SkeletonDisplayManager class:

void GetCoordinates(JointType jointType, IEnumerable<Joint> joints, out float x, out float y)
{
    var joint = joints.First(j => j.JointType == jointType);

    Vector2 vector2 = Convert(kinectSensor, joint.Position);

    x = (float)(vector2.X * rootCanvas.ActualWidth);
    y = (float)(vector2.Y * rootCanvas.ActualHeight);
}

With an absolute coordinate, it is easy to deduce the canvas space coordinate of the joint:

x = (float)(vector2.X * rootCanvas.ActualWidth);
y = (float)(vector2.Y * rootCanvas.ActualHeight);

Finally, with the help of the previous methods, the Plot and Trace methods are defined as follows:

void Plot(JointType centerID, IEnumerable<Joint> joints)
{
    float centerX;
    float centerY;

    GetCoordinates(centerID, joints, out centerX, out centerY);

    const double diameter = 8;

    Ellipse ellipse = new Ellipse
    {
        Width = diameter,
        Height = diameter,
        HorizontalAlignment = HorizontalAlignment.Left,
        VerticalAlignment = VerticalAlignment.Top,
        StrokeThickness = 4.0,
        Stroke = new SolidColorBrush(Colors.Green),
        StrokeLineJoin = PenLineJoin.Round
    };

    Canvas.SetLeft(ellipse, centerX - ellipse.Width / 2);
    Canvas.SetTop(ellipse, centerY - ellipse.Height / 2);

    rootCanvas.Children.Add(ellipse);
}

void Trace(JointType sourceID, JointType destinationID, JointCollection joints)
{
    float sourceX;
    float sourceY;

    GetCoordinates(sourceID, joints, out sourceX, out sourceY);

    float destinationX;
    float destinationY;

    GetCoordinates(destinationID, joints, out destinationX, out destinationY);

    Line line = new Line
                    {
                        X1 = sourceX,
                        Y1 = sourceY,
                        X2 = destinationX,
                        Y2 = destinationY,
                        HorizontalAlignment = HorizontalAlignment.Left,
                        VerticalAlignment = VerticalAlignment.Top,
                        StrokeThickness = 4.0,
                        Stroke = new SolidColorBrush(Colors.Green),
                        StrokeLineJoin = PenLineJoin.Round
                    };


    rootCanvas.Children.Add(line);
}

The main point to remember here is that WPF shapes (Line or Ellipse) are created to represent parts of the skeleton. After the shape is created, it is added to the canvas.

The only specific joint in the skeleton is the head because it makes sense to draw it bigger than the other joints to represent the head of the skeleton. To do so, a new Plot method is defined:

void Plot(JointType centerID, JointType baseID, JointCollection joints)
{
    float centerX;
    float centerY;

    GetCoordinates(centerID, joints, out centerX, out centerY);

    float baseX;
    float baseY;

    GetCoordinates(baseID, joints, out baseX, out baseY);

    double diameter = Math.Abs(baseY - centerY);

    Ellipse ellipse = new Ellipse
    {
        Width = diameter,
        Height = diameter,
        HorizontalAlignment = HorizontalAlignment.Left,
        VerticalAlignment = VerticalAlignment.Top,
        StrokeThickness = 4.0,
        Stroke = new SolidColorBrush(Colors.Green),
        StrokeLineJoin = PenLineJoin.Round
    };

    Canvas.SetLeft(ellipse, centerX - ellipse.Width / 2);
    Canvas.SetTop(ellipse, centerY - ellipse.Height / 2);

    rootCanvas.Children.Add(ellipse);
}

In this case, the ellipse’s diameter is defined using the distance between the head and the center of shoulder.

Finally, you can also add a new parameter to the Draw method to support the seated mode. In this case, you must not draw the lower body joints:

public void Draw(Skeleton[] skeletons, bool seated)
{
    rootCanvas.Children.Clear();
    foreach (Skeleton skeleton in skeletons)
    {
        if (skeleton.TrackingState != SkeletonTrackingState.Tracked)
            continue;

        Plot(JointType.HandLeft, skeleton.Joints);
        Trace(JointType.HandLeft, JointType.WristLeft, skeleton.Joints);
        Plot(JointType.WristLeft, skeleton.Joints);
        Trace(JointType.WristLeft, JointType.ElbowLeft, skeleton.Joints);
        Plot(JointType.ElbowLeft, skeleton.Joints);
        Trace(JointType.ElbowLeft, JointType.ShoulderLeft, skeleton.Joints);
        Plot(JointType.ShoulderLeft, skeleton.Joints);
        Trace(JointType.ShoulderLeft, JointType.ShoulderCenter, skeleton.Joints);
        Plot(JointType.ShoulderCenter, skeleton.Joints);

        Trace(JointType.ShoulderCenter, JointType.Head, skeleton.Joints);

        Plot(JointType.Head, JointType.ShoulderCenter, skeleton.Joints);

        Trace(JointType.ShoulderCenter, JointType.ShoulderRight, skeleton.Joints);
        Plot(JointType.ShoulderRight, skeleton.Joints);
        Trace(JointType.ShoulderRight, JointType.ElbowRight, skeleton.Joints);
        Plot(JointType.ElbowRight, skeleton.Joints);
        Trace(JointType.ElbowRight, JointType.WristRight, skeleton.Joints);
        Plot(JointType.WristRight, skeleton.Joints);
        Trace(JointType.WristRight, JointType.HandRight, skeleton.Joints);
        Plot(JointType.HandRight, skeleton.Joints);

        if (!seated)
        {
            Trace(JointType.ShoulderCenter, JointType.Spine, skeleton.Joints);
            Plot(JointType.Spine, skeleton.Joints);
            Trace(JointType.Spine, JointType.HipCenter, skeleton.Joints);
            Plot(JointType.HipCenter, skeleton.Joints);

            Trace(JointType.HipCenter, JointType.HipLeft, skeleton.Joints);
            Plot(JointType.HipLeft, skeleton.Joints);
            Trace(JointType.HipLeft, JointType.KneeLeft, skeleton.Joints);
            Plot(JointType.KneeLeft, skeleton.Joints);
            Trace(JointType.KneeLeft, JointType.AnkleLeft, skeleton.Joints);
            Plot(JointType.AnkleLeft, skeleton.Joints);
            Trace(JointType.AnkleLeft, JointType.FootLeft, skeleton.Joints);
            Plot(JointType.FootLeft, skeleton.Joints);

            Trace(JointType.HipCenter, JointType.HipRight, skeleton.Joints);
            Plot(JointType.HipRight, skeleton.Joints);
            Trace(JointType.HipRight, JointType.KneeRight, skeleton.Joints);
            Plot(JointType.KneeRight, skeleton.Joints);
            Trace(JointType.KneeRight, JointType.AnkleRight, skeleton.Joints);
            Plot(JointType.AnkleRight, skeleton.Joints);
            Trace(JointType.AnkleRight, JointType.FootRight, skeleton.Joints);
            Plot(JointType.FootRight, skeleton.Joints);
        }
    }
}