Kinect SDK 1.0 - 2 - Use the ColorStream

19. April 2012 16:13 by Renaud in   //  Tags:   //   Comments (1)
 1. Introduction to the API
 2. Use the ColorImageStream
 3. Track the users with the SkeletonStream
 4. Kinect in depth!
 5. Speech recognition

In the first blog post talking about Kinect,  I introduced to you the Kinect SDK 1.0. For that second part, I'd like to talk about the ColorStream. That stream will allow us to get and display a video from the Kinect. At the end of this article, you will know how to realize a simple WPF application to:

  • Display the video from the Kinect camera
  • Allow the user to select the image format
  • Take a snapshot and save it on your hard drive
  • Transform the video image in real-time

If you want to try it, you can download this sample project :

What is the ColorStream?

This is the data stream that will allow us to retrieve what the Kinect sees. There are 3 main streams we can play with:

  • ColorStream: gives access to the color image.
  • DepthStream: gives information about the distance between the Kinect and a point in the space.
  • SkeletonStream: gives information about people standing in front of the Kinect sensor.

Basically, all those streams are used in the same way (because they all inherit from the same base classe: ImageStream). We will talk about the two others in the next blog posts, but for now, let's focus on the simplest one: the ColorStream.

Initialize the stream

To start a stream (and thus start receiving information), we have to activate it explictly. In the code below, we enable the ColorStream with a given ColorImageFormat:

        /// <summary>
        /// Opens the kinect.
        /// </summary>
        /// <param name="newKinect">The new kinect.</param>
        private void OpenKinect(KinectSensor newKinect)
        {
            // Enables the ColorStream with the default format
            newKinect.ColorStream.Enable(ColorImageFormat.RgbResolution640x480Fps30);
            // Listen to the ColorFrameReady event to know when data is available
            newKinect.ColorFrameReady += newKinect_ColorFrameReady;
            // Starts the kinect sensor
            newKinect.Start();
        }

Then we will have to process the data!

Data processing

We need some fields to store the useful information:

        /// <summary>
        /// The length of the pixels data
        /// </summary>
        private int pixelDataLength;

        /// <summary>
        /// this array will contain the data provided by the Kinect
        /// </summary>
        private byte[] pixelData;

        /// <summary>
        /// Indicates wether the selected format is supported or not
        /// </summary>
        private bool isFormatSupported;

        /// <summary>
        /// the zone to update in the writeableBitmap
        /// </summary>
        private Int32Rect int32Rect;

        /// <summary>
        /// The amounts of bytes for one row
        /// </summary>
        private int stride;

        /// <summary>
        /// Gets or sets the output image.
        /// </summary>
        /// <value>
        /// The output image.
        /// </value>
        public WriteableBitmap OutputImage { get; set; }

Then, we implement the ColorFrameReady event handler:

        /// <summary>
        /// Handles the ColorFrameReady event of the newKinect control.
        /// </summary>
        /// <param name="sender">The source of the event.</param>
        /// <param name="e">The <see cref="Microsoft.Kinect.ColorImageFrameReadyEventArgs"/> instance containing the event data.</param>
        void newKinect_ColorFrameReady(object sender, ColorImageFrameReadyEventArgs e)
        {
            // Add some code here
        }

First, we attempt to open a ColorImageFrame. It may happen that the frame is null, for example when it's open too much time after the event is raised, or when the Kinect is stopped.

            // Open the received frame
            using (ColorImageFrame colorFrame = e.OpenColorImageFrame())
            {
                // Could be null if we opened it too late
                if (colorFrame == null)
                     return;

                // Add some code here
            }

If the frame isn't null, then we will compare the data size with the pixelDataLength private field (which should be 0 at the beginning).

If the sizes are different, it means that we are going through that method for the first time, so we will initialize some of the objects based on the ColorImageFormat used to enable the stream. We will create a byte array with the right size (to calculate the size, you can use some properties of the ColorImageFrame: the array should have a length equals to FrameWidth x FrameHeigth x FrameBytesPerPixel). Then we will also create a WriteableBitmap. This is just like a BitmapImage, except that you can update the pixels of the region you want at any time, and it will update the UI automatically (exactly what we need in this application). Moreover it keeps us from instantiating a new BitmapImage 30 times per second.

                    // Checks if the length has changed (means the format has changed)
                    if (pixelDataLength != colorFrame.PixelDataLength)
                    {
                        // creates a new buffer long enough to receives all the data of a frame
                        pixelData = new byte[colorFrame.PixelDataLength];
                        pixelDataLength = colorFrame.PixelDataLength;

                        // Use a WriteableBitmap because it's better to re-write some pixels
                        // of a WriteabeBitmap than creating a new BitmapImage for each new frame.
                        OutputImage = new WriteableBitmap(
                                             colorFrame.Width,
                                             colorFrame.Height,
                                             96, // Standard dpi
                                             96,
                             // All the formats provided by the kinect are 4-bytes per pixel 
                             // (except the RawYuv which is not supported)
                                             PixelFormats.Bgr32,
                                             null);

                        // The rectangle that we will update in the WriteableBitmap 
                        // (has same dimensions as OutputImage)
                        int32Rect = new Int32Rect(0, 0, colorFrame.Width, colorFrame.Height);

                        // how many bytes we need to represent one pixel
                        stride = colorFrame.Width * colorFrame.BytesPerPixel;

                        // Set the new WriteableBitmap as the KinectImage source
                        KinectImage.Source = OutputImage;
                    }

KinectImage is an Image control that you have to define in the XAML part of the MainWindow class:

 <Image x:Name="KinectImage" Width="640" Height="480" />

Update the image

So now we have a WriteableBitmap ready to receive our data, and we just have to update it each time we receive a new frame.

The CopyPixelDataTo method will copy the content of the frame to the given byte array. Then we have to update the OutputImage:

                    // Copies the data from the fram to the pixelData array.
                    colorFrame.CopyPixelDataTo(pixelData);

                    // Update the writeable bitmap.
                    OutputImage.WritePixels(
                        // Zone that will be updated
                        int32Rect,
                        // new data
                        pixelData,
                        // stride = number of pixels for one line of the image
                        stride,
                        // starting index = 0
                        0);

You can run the application now, and see the result! :)

Change the format !

At any time, you can see the current ColorImageFormat using the property Format of the ColorStream. That value gives two informations:

  • How is represented each pixel
  • The amount of frames per second
We need to add a combobox to the UI.
            <ComboBox Width="200" x:Name="FormatComboBox" SelectionChanged="FormatComboBox_SelectionChanged" />

Then in the constructor of MainWindows, we initialize the combobox values :

            // Fill in the Format Combobox with the available formats
            IList<ColorImageFormat> colorImageFormat = new List<ColorImageFormat> 
                                                        { ColorImageFormat.RgbResolution640x480Fps30,
                                                        ColorImageFormat.RgbResolution1280x960Fps12,
                                                        ColorImageFormat.RawYuvResolution640x480Fps15,
                                                        ColorImageFormat.YuvResolution640x480Fps15
                                                        };

            // Populate the combobox with the new List
            FormatComboBox.ItemsSource = colorImageFormat;

And finally, we implement the FormatComboBox_SelectionChanged eventhandler. It's not possible to modify directly the Format property because it is read-only. To change the format, you do have to pass it as a parameter in the Enable(...) method.

        private void FormatComboBox_SelectionChanged(object sender, SelectionChangedEventArgs e)
        {
            if (e.AddedItems.Count > 0
                && Kinect != null
                && Kinect.Status == KinectStatus.Connected)
            {
                // Disabled the current stream
                Kinect.ColorStream.Disable();
                // Retrieves the new format
                var newFormat = (ColorImageFormat)e.AddedItems[0];

                // Enable the stream with the new format
                Kinect.ColorStream.Enable(newFormat);
            }
        }

When the stream is re-enabled, the ColorFrameReady fires again, and the image is updated.

Take a picture!

A nice feature to have in your Kinect application is a the possibility to take a picture!

There are different ways to achieve this, and I'm going to show you two possibilities: an easy one, and a very easy one!

First, let's add a new button to start the snapshot task:

 <Button Content="Take a picture!" x:Name="PictureTask" Click="PictureTask_Click" />

Then, implement the button click event handler. We open a new SaveDialog to ask for where to save the picture. If a file already exists at the given path, then it will be overwritten.

        /// <summary>
        /// Handles the Click event of the PictureTask button.
        /// </summary>
        /// <param name="sender">The source of the event.</param>
        /// <param name="e">The <see cref="System.Windows.RoutedEventArgs"/> instance containing the event data.</param>
        private void PictureTask_Click(object sender, RoutedEventArgs e)
        {
            string fileName = null;
            SaveFileDialog saveDialog = new SaveFileDialog();
            if (saveDialog.ShowDialog().Value)
            {
                fileName = saveDialog.FileName;
            }

            if (string.IsNullOrWhiteSpace(fileName))
                return;

            if (File.Exists(fileName))
            {
                File.Delete(fileName);
            }

            // Add some logic to take a picture here..

        }

Now you need to add the logic to take a picture and save it to your hard drive. Here are the two solutions that I propose you:

The easy way
            using (FileStream savedSnapshot = new FileStream(fileName, FileMode.CreateNew))
            {
                BitmapSource image = (BitmapSource)KinectImage.Source;
                JpegBitmapEncoder jpgEncoder = new JpegBitmapEncoder();
                jpgEncoder.QualityLevel = 70;
                jpgEncoder.Frames.Add(BitmapFrame.Create(image));
                jpgEncoder.Save(savedSnapshot);
                savedSnapshot.Flush();
                savedSnapshot.Close();
                savedSnapshot.Dispose();
            }
The even easier way (Coding4Fun Toolkit)

Add to your project a reference to the Coding4Fun's Kinect Toolkit:

[caption id="attachment_1709" align="aligncenter" width="635" caption="Add a reference to the toolkit using NuGet"][/caption]

Add a new using instruction to be able to use the extension methods:

using Coding4Fun.Kinect.Wpf;

And finally, you just need those two lines:

            // Coding4Fun helper's method:
            BitmapSource image = (BitmapSource)KinectImage.Source;
            image.Save(fileName, ImageFormat.Jpeg);

The Kinect Toolkit allows you to call a Save method on a BitmapSource and passing two parameters: the file path, and the outpout format.

Transform the output image

The final part of this article will talk about how you can manipulate the color image frame before you display it. As you know, the color frame is nothing more than a bunch of bytes stored in an array called pixelData.

Let's take, say, the ColorImageFormat.RgbResolution640x480Fps30:

pixelData contains 640 x 480 x 4 = 1228800 bytes. It means 4 bytes for each pixel in the  color frame. The first three bytes correspond to the colors Blue, Green, Red. The fourth one isn't used but could store the alpha (transparency) channel in a Bgra format.

If we simply try to invert the bytes values before we update the WriteableBitmap, it will result in a "negative" effect!

                    for (int i = 0; i < pixelData.Length - 3; i +=4)
                    {
                        // Transformation 1:
                        // ======================================= //
                        // ( ~ ) inverts the bits 
                        pixelData[i] = (byte)~pixelData[i];
                        pixelData[i + 1] = (byte)~pixelData[i + 1];
                        pixelData[i + 2] = (byte)~pixelData[i + 2];
                    }
The tild  (~) operator is used to invert each bit in a numeric value.

You can now easily imagine a lot of transformations. You could set the Blue and Green values to 0 to display a Red image. To create a gray scale, each byte (B, G, and R) must have the same value.

Thank you for reading! In the next post, we will talk about the SkeletonStream!

 

Comments (1) -

Filiberto Schwebach
Filiberto Schwebach
4/14/2013 5:00:37 PM #

Thanks the website was great for what I need

Pingbacks and trackbacks (1)+

Add comment

  Country flag

biuquote
Loading

TextBox

About the author

I'm a developer, blog writer, and author, mainly focused on Microsoft technologies (but not only Smile). I'm Microsoft MVP Client Development since July 2013.

Microsoft Certified Professional

I'm currently working as an IT Evangelist with an awesome team at the Microsoft Innovation Center Belgique, where I spend time and energy helping people to develop their projects. I also give training to enthusiastic developers and organize afterworks with the help of the Belgian community.

MIC Belgique

Take a look at my first book (french only): Développez en HTML 5 pour Windows 8

Développez en HTML5 pour Windows 8

Membre de l'association Fier d'être développeur

TextBox

Month List