Kinect SDK 1.0 - 3 - Track bodies with the SkeletonStream

19. April 2012 16:10 by Renaud in   //  Tags:   //   Comments (0)
 1. Introduction to the API
 2. Use the ColorImageStream
 3. Track the users with the SkeletonStream
 4. Kinect in depth!
 5. Speech recognition

In the previous posts, we saw how to connect and use your Kinect within your app, and how to use the ColorStream. That was just enough to use your Kinect as a webcam, but not much more.

In this post, we are going to empower your application with the Skeleton tracking provided by the Kinect !

What is the SkeletonStream ?

This is a data stream provided by the Kinect which allows you to know the position of a user in front of the Kinect, and even to know the positions of up to 20 Joints on the user's body.

 

The following figure shows you the Joints (yellow points) tracked by the Kinect in the 3-dimensional referential.

The skeleton of a user and all the tracked points are stored in an object called Skeleton (MSDN).

Skeleton in details

The Skeleton doesn't expose a lot of properties, but they are very useful :

  • TrackingState : The status tells you whether the Skeleton is tracked (Tracked) or not (NotTracked), or if we only have the global position of the user (PositionOnly). The Kinect is able to follow up to 6 users at the same time: 2 in Tracked mode, and 4 in PositionOnly mode. The Tracked mode gives you access to the full skeleton and the 20 Joints.
  • Joints : A Joint collection representing the body as shown on the above image (only available if Tracked).
  • TrackingId :  Identifies a Skeleton, so that you can track a particular user over the time.
  • Position : The global position of a user (available if Tracked or PositionOnly).
  • ClippedEdges : Indicates whether one part of the user's body is clipped by the edges (not in the field of view).

It's important to know that all the positions are represented by a SkeletonPosition object which has 3 properties: X, Y, Z. The distances are expressed in meters and the referential is as shown in the above schema: X goes from left to right, Y goes from up to down and Z is the depth. So basically, the Kinect is the point (0, 0, 0).

[caption id="attachment_126" align="aligncenter" width="92" caption="WpfKinect 3 - SkeletonStream"][/caption]

Track and draw the skeleton !

What we do want now is to use that SkeletonStream. We will first activate it, as we did for the ColorStream in the previous post :

/// <summary>
/// This array will store the Skeletons
/// received with each SkeletonFrame
/// </summary>
private Skeleton[] skeletonData = new Skeleton[6];

/// <summary>
/// Starts the Kinect sensor.
/// </summary>
/// <param name="newKinect">Le nouveau capteur.</param>
private void OpenKinect(KinectSensor newKinect)
{
    Kinect = newKinect;
    // Activate the SkeletonStream with default smoothing.
    newKinect.SkeletonStream.Enable();
    // Subscribe to the SkeletonFrameReady event to know when data is available
    newKinect.SkeletonFrameReady += Kinect_SkeletonFrameReady;
    // Starts the sensor
    newKinect.Start();
}

Then you implement the eventhandler :

        void Kinect_SkeletonFrameReady(object sender, SkeletonFrameReadyEventArgs e)
        {
            // Opens the received SkeletonFrame
            using (SkeletonFrame skeletonFrame = e.OpenSkeletonFrame())
            {
                // Skeleton frame might be null if we are too late or if the Kinect has been stopped
                if (skeletonFrame == null)
                    return;

                // Copies the data in a Skeleton array (6 items) 
                skeletonFrame.CopySkeletonDataTo(skeletonData);

                // TODO : Do something with the skeletons

            }
        }

Now you can start processing the information using the received Skeletons. We will start by drawing the skeletons of the two tracked users (if any).

To make it properly, I created a SkeletonDrawing helper class, which will allow me to draw a skeleton in a canvas based on the Joints. You can see the source code here: SkeletonDrawing.cs

After the todo in the Kinect_SkeletonFrameReady eventhandler, we will add a a few lines of code to retrieve the two tracked skeletons and update the corresponding SkeletonDrawing object.

First, add a dictionary to keep track of the drawings:

        // The Skeletons drawing on the scene
        private Dictionary<int, SkeletonDrawing> drawnSkeletons;

Then, add the logic to look for the tracked skeletons and update them:

                // Retrieves Skeleton objects with Tracked state
                var trackedSkeletons = skeletonData.Where(s => s.TrackingState == SkeletonTrackingState.Tracked);

                // By default, assume all the drawn skeletons are inactive
                foreach (SkeletonDrawing skeleton in drawnSkeletons.Values)
                    skeleton.Status = ActivityState.Inactive;

                foreach (Skeleton trackedSkeleton in trackedSkeletons)
                {
                    SkeletonDrawing skeletonDrawing;
                    // Checks if the tracked skeleton is already drawn.
                    if (!drawnSkeletons.TryGetValue(trackedSkeleton.TrackingId, out skeletonDrawing))
                    {
                        // If not, create a new drawing on our canvas
                        skeletonDrawing = new SkeletonDrawing(this.SkeletonCanvas);
                        drawnSkeletons.Add(trackedSkeleton.TrackingId, skeletonDrawing);
                    }

                    // Update the drawing
                    Update(trackedSkeleton, skeletonDrawing);
                    skeletonDrawing.Status = ActivityState.Active;
                }

                foreach (SkeletonDrawing skeleton in drawnSkeletons.Values)
                {
                    // Erase all the still inactive drawings. It means they are not tracked anymore.
                    if (skeleton.Status == ActivityState.Inactive)
                        skeleton.Erase();
                }

Then, the update method, which is going to make some transformations on the SkeletonPoint positions and update the drawn skeleton :

        /// <summary>
        /// Updates the specified drawn skeleton with the new positions
        /// </summary>
        /// <param name="skeleton">The skeleton source.</param>
        /// <param name="drawing">The target drawing.</param>
        private void Update(Skeleton skeleton, SkeletonDrawing drawing)
        {
            foreach (Joint joint in skeleton.Joints)
            {
                // Transforms a SkeletonPoint to a ColorImagePoint
                var colorPoint = Kinect.MapSkeletonPointToColor(joint.Position, Kinect.ColorStream.Format);
                // Scale the ColorImagePoint position to the current window size
                var point = new Point((int)colorPoint.X / 640.0 * this.ActualWidth, (int)colorPoint.Y / 480.0 * this.ActualHeight);
                // update the position of that joint
                drawing.Update(joint.JointType, point);
            }
        }

Notice the MapSkeletonPointToColor method!  The KinectSensor class provides some helper methods to map a point from a stream to another.

Here are all the available methods:

The reason why those methods exist is simple:

The SkeletonStream gives information in a different referential than the ColorStream and the DepthStream. For the last two, you have a 2D position for each pixel of the frame, with the top-left corner as the origin (0,0).

The SkeletonPoints are placed in a 3D referential. Now if you want to display the ColorStream, and overlay with the Skeleton, you'll have to convert the skeleton positions to make sure they fit with the ColorStream image. The MapSkeletonPointToColor does the job for you and give you a ColorImagePoint based on a SkeletonPoint!

The case of the depth and color points is a little bit different. The depth sensor is not at the exact same place than the color sensor. It means that there is a small gap between the two frames (we will see it in the next post!).

Until now, we didn't use the Z value but we will see how we can add it to scale the size of the shapes, and show the proximity.

Postures and Gestures

Let's talk about the gestures. There is nothing in the SDK to easily detect gestures, and you'll have to implement everything by yourself! There are some projects out there to help you achieve that job, but it's still complicated.

If you plan to use postures or gestures in your application, you should definitely check the KinectToolbox project on codeplex:

http://kinecttoolbox.codeplex.com/

Pingbacks and trackbacks (1)+

Add comment

  Country flag

biuquote
Loading

TextBox

About the author

I'm a developer, blog writer, and author, mainly focused on Microsoft technologies (but not only Smile). I'm Microsoft MVP Client Development since July 2013.

Microsoft Certified Professional

I'm currently working as an IT Evangelist with an awesome team at the Microsoft Innovation Center Belgique, where I spend time and energy helping people to develop their projects. I also give training to enthusiastic developers and organize afterworks with the help of the Belgian community.

MIC Belgique

Take a look at my first book (french only): Développez en HTML 5 pour Windows 8

Développez en HTML5 pour Windows 8

Membre de l'association Fier d'être développeur

TextBox

Month List