Kinect Advance Programming Session at HEC Islamabad

today I gave a session at Higher Education commission Pakistan about Kinect Programming. The agenda was to use common features in programming Kinect app like, ColorImageFrame, DepthImageFrame, SkeletonFrame, Skeleton tracking, Making gestures, Speech recognition, see device capabilities, possible applications. The overall response was great and I saw people interested in making apps using this device in future.

Universities attending the session were,

University ok Karachi
Virtual University
Sindh Agriculture University
Govt. Collage Uni Lahore
Lahore collage for Women University
IBA Karachi

Please find the Kinect Session Slides below:

Kinect Session Slides

Basics of Kinect – Demo App

In this app, we will do ColorImageFrame, Skeletal Tracking, add a normal gesture that if you move your hand above your head it will show a message to user. We will also see how to use depth Frame as well.

UI CODE:
///////






    
        

            
            
            
            
            
            
        
    





CODE BEHIND:
///////////




using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Windows;
using System.Windows.Controls;
using System.Windows.Data;
using System.Windows.Documents;
using System.Windows.Input;
using System.Windows.Media;
using System.Windows.Media.Imaging;
using System.Windows.Navigation;
using System.Windows.Shapes;
using Microsoft.Kinect;
using Microsoft.Kinect.Toolkit;
using Microsoft.Kinect.Toolkit.FaceTracking;

using System.Windows.Media;
using System.Windows.Media.Imaging;
namespace KinectDemo1
{
    /// 
    /// Interaction logic for MainWindow.xaml
    /// 
    public partial class MainWindow : Window
    {
        private static readonly int Bgr32BytesPerPixel = (PixelFormats.Bgr32.BitsPerPixel + 7) / 8;
        private readonly KinectSensorChooser sensorChooser = new KinectSensorChooser();
        private WriteableBitmap colorImageWritableBitmap;
        private byte[] colorImageData;
        private ColorImageFormat currentColorImageFormat = ColorImageFormat.Undefined;
        bool closing = false;
        const int skeletonCount = 6;
        Skeleton[] allSkeletons = new Skeleton[skeletonCount];


        public MainWindow()
        {
            InitializeComponent();

            sensorChooser.KinectChanged += sensorChooser_KinectChanged;

            sensorChooser.Start();
            
        }

        void _sensor_AllFramesReady(object sender, AllFramesReadyEventArgs e)
        {
            if (closing)
            {
                return;
            }
            
           

            
        }


        void sensorChooser_KinectChanged(object sender, KinectChangedEventArgs e)
        {
            KinectSensor oldSensor = e.OldSensor;
            StopKinect(oldSensor);


            KinectSensor newSensor = e.NewSensor;


            var parameters = new TransformSmoothParameters
            {
                Smoothing = 0.3f,
                Correction = 0.0f,
                Prediction = 0.0f,
                JitterRadius = 1.0f,
                MaxDeviationRadius = 0.5f
            };

            this.colorPixels = new byte[newSensor.ColorStream.FramePixelDataLength];
            this.colorPixelsDepth = new byte[newSensor.ColorStream.FramePixelDataLength];
            // This is the bitmap we'll display on-screen
            this.colorBitmap = new WriteableBitmap(newSensor.ColorStream.FrameWidth, newSensor.ColorStream.FrameHeight, 96.0, 96.0, PixelFormats.Bgr32, null);
            this.colorBitmapDepth = new WriteableBitmap(newSensor.ColorStream.FrameWidth, newSensor.ColorStream.FrameHeight, 96.0, 96.0, PixelFormats.Bgr32, null);
            
            newSensor.AllFramesReady += new EventHandler(_sensor_AllFramesReady);
            newSensor.ColorFrameReady += newSensor_ColorFrameReady;
            newSensor.SkeletonFrameReady += newSensor_SkeletonFrameReady;
            newSensor.DepthFrameReady += newSensor_DepthFrameReady;
            newSensor.ColorStream.Enable(ColorImageFormat.RgbResolution640x480Fps30);
            newSensor.DepthStream.Enable(DepthImageFormat.Resolution640x480Fps30);
            
            newSensor.SkeletonStream.Enable(parameters);


            try
            {
                newSensor.Start();
            }
            catch (System.IO.IOException)
            {
                sensorChooser.TryResolveConflict();


            }
        }

        void newSensor_DepthFrameReady(object sender, DepthImageFrameReadyEventArgs e)
        {
            DepthImageFrame imageFrame = e.OpenDepthImageFrame();
                if (imageFrame != null)
                {
                    short[] pixelData =  new short[imageFrame.PixelDataLength];
                    imageFrame.CopyPixelDataTo(pixelData);
                
                int[] depth = new int[
                    imageFrame.PixelDataLength];
                int[] player = new int[
                                imageFrame.PixelDataLength];

                for (int i = 0; i < depth.Length; i++)
                {
                    player[i] = pixelData[i] &
                           DepthImageFrame.PlayerIndexBitmask;
                    depth[i] = ((ushort)pixelData[i]) >>
                        DepthImageFrame.PlayerIndexBitmaskWidth;
                }
                this.colorBitmapDepth.WritePixels(
                            new Int32Rect(0, 0, this.colorBitmapDepth.PixelWidth, this.colorBitmapDepth.PixelHeight),
                            this.colorPixelsDepth,
                            this.colorBitmapDepth.PixelWidth * sizeof(int),
                            0);
                imgKinectDepth.Source = colorBitmapDepth;
                }
        }

        

        void newSensor_SkeletonFrameReady(object sender, SkeletonFrameReadyEventArgs e)
        {
            using (SkeletonFrame skeletonFrameData = e.OpenSkeletonFrame())
            {
                if (skeletonFrameData != null)
                {
                    
                

                skeletonFrameData.CopySkeletonDataTo(allSkeletons);

                Skeleton first = (from s in allSkeletons
                                  where s.TrackingState == SkeletonTrackingState.Tracked
                                  select s).FirstOrDefault();
                    if (first != null)
                    {
                        SetEllipsePosition(Head, first.Joints[JointType.Head]);
                        SetEllipsePosition(leftHand, first.Joints[JointType.HandLeft]);
                        SetEllipsePosition(rightHand, first.Joints[JointType.HandRight]);
                        ProcessHandsUpGesture(first.Joints[JointType.Head], first.Joints[JointType.HandLeft], first.Joints[JointType.HandRight]);

                    }
                }
            }
        }
        /// 
        /// Bitmap that will hold color information
        /// 
        private WriteableBitmap colorBitmap;
        private WriteableBitmap colorBitmapDepth;
        /// 
        /// Intermediate storage for the color data received from the camera
        /// 
        private byte[] colorPixels;
        private byte[] colorPixelsDepth;

        void newSensor_ColorFrameReady(object sender, ColorImageFrameReadyEventArgs e)
        {

            using (ColorImageFrame colorFrame = e.OpenColorImageFrame())
            {
                if (colorFrame != null)
                {
                    // Copy the pixel data from the image to a temporary array
                    colorFrame.CopyPixelDataTo(this.colorPixels);

                    // Write the pixel data into our bitmap
                    this.colorBitmap.WritePixels(
                        new Int32Rect(0, 0, this.colorBitmap.PixelWidth, this.colorBitmap.PixelHeight),
                        this.colorPixels,
                        this.colorBitmap.PixelWidth * sizeof(int),
                        0);
                }
            }
            imgKinect.Source = colorBitmap;

        }


        void StopKinect(KinectSensor sensor)
        {
            if (sensor != null)
            {
                sensor.Stop();
                sensor.AudioSource.Stop();

            }
        }

        private void ProcessHandsUpGesture(Joint head, Joint handleft, Joint handright)
        {
            if (handright.Position.Y > head.Position.Y)
            {
                this.Title = "Your Hand Is Above Your Head";
            }
        }

        private void SetEllipsePosition(Ellipse ellipse, Joint joint)
        {
            SkeletonPoint vector = new SkeletonPoint();
            vector.X = ScaleVector(640, joint.Position.X);
            vector.Y = ScaleVector(480, -joint.Position.Y);
            vector.Z = joint.Position.Z;

            Joint updatedJoint = new Joint();
            //updatedJoint = joint.ID;
            updatedJoint.TrackingState = JointTrackingState.Tracked;
            updatedJoint.Position = vector;

            Canvas.SetLeft(ellipse, updatedJoint.Position.X);
            Canvas.SetTop(ellipse, updatedJoint.Position.Y);


        }

        private float ScaleVector(int length, float position)
        {
            float value = (((((float)length) / 1f) / 2f) * position) + (length / 2);
            if (value > length)
            {
                return (float)length;
            }
            if (value < 0f)
            {
                return 0f;
            }
            return value;
        }


        private void Window_Closing(object sender, System.ComponentModel.CancelEventArgs e)
        {
            StopKinect(sensorChooser.Kinect);
        }


    }
}




Basics of Kinect – How to recieve ColorImageFrame from Kinect device.

The goal of the code below is to connect to Kinect device, and load the video stream to an image control that is placed on the UI. The name of image control is “imgKinect”.

UI CODE
=======
This is some source code:




    
        

            
            
            
            
            
            
        
    


Code Behind
===========


using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Windows;
using System.Windows.Controls;
using System.Windows.Data;
using System.Windows.Documents;
using System.Windows.Input;
using System.Windows.Media;
using System.Windows.Media.Imaging;
using System.Windows.Navigation;
using System.Windows.Shapes;
using Microsoft.Kinect;
using Microsoft.Kinect.Toolkit;

using System.Windows.Media;
using System.Windows.Media.Imaging;
namespace KinectDemo1
{
    /// 
    /// Interaction logic for MainWindow.xaml
    /// 
    public partial class MainWindow : Window
    {
        private static readonly int Bgr32BytesPerPixel = (PixelFormats.Bgr32.BitsPerPixel + 7) / 8;
        private readonly KinectSensorChooser sensorChooser = new KinectSensorChooser();
        private WriteableBitmap colorImageWritableBitmap;
        private byte[] colorImageData;
        private ColorImageFormat currentColorImageFormat = ColorImageFormat.Undefined;
        bool closing = false;

        public MainWindow()
        {
            InitializeComponent();

            sensorChooser.KinectChanged += sensorChooser_KinectChanged;
            sensorChooser.Start();
           
        }

        void _sensor_AllFramesReady(object sender, AllFramesReadyEventArgs e)
        {
            if (closing)
            {
                return;
            }      
        }

        void sensorChooser_KinectChanged(object sender, KinectChangedEventArgs e)
        {
            KinectSensor oldSensor = e.OldSensor;
            StopKinect(oldSensor);

            KinectSensor newSensor = e.NewSensor;

            this.colorPixels = new byte[newSensor.ColorStream.FramePixelDataLength];

            // This is the bitmap we'll display on-screen
            this.colorBitmap = new WriteableBitmap(newSensor.ColorStream.FrameWidth, newSensor.ColorStream.FrameHeight, 96.0, 96.0, PixelFormats.Bgr32, null);
            this.colorBitmapDepth = new WriteableBitmap(newSensor.ColorStream.FrameWidth, newSensor.ColorStream.FrameHeight, 96.0, 96.0, PixelFormats.Bgr32, null);
            
            newSensor.AllFramesReady += new EventHandler<allframesreadyeventargs>(_sensor_AllFramesReady);
            newSensor.ColorFrameReady += newSensor_ColorFrameReady;

            newSensor.ColorStream.Enable(ColorImageFormat.RgbResolution640x480Fps30);

            try
            {
                newSensor.Start();
            }
            catch (System.IO.IOException)
            {
                sensorChooser.TryResolveConflict();
            }
        }
                
        private WriteableBitmap colorBitmap;

        private byte[] colorPixels;

        void newSensor_ColorFrameReady(object sender, ColorImageFrameReadyEventArgs e)
        {

            using (ColorImageFrame colorFrame = e.OpenColorImageFrame())
            {
                if (colorFrame != null)
                {
                    // Copy the pixel data from the image to a temporary array
                    colorFrame.CopyPixelDataTo(this.colorPixels);

                    // Write the pixel data into our bitmap
                    this.colorBitmap.WritePixels(
                        new Int32Rect(0, 0, this.colorBitmap.PixelWidth, this.colorBitmap.PixelHeight),
                        this.colorPixels,
                        this.colorBitmap.PixelWidth * sizeof(int),
                        0);
                }
            }
            imgKinect.Source = colorBitmap;

        }

        void StopKinect(KinectSensor sensor)
        {
            if (sensor != null)
            {
                sensor.Stop();
                sensor.AudioSource.Stop();
            }
        }       

        private void Window_Closing(object sender, System.ComponentModel.CancelEventArgs e)
        {
            StopKinect(sensorChooser.Kinect);
        }

    }
}

Basics of Kinect – How to initialize Kinect Device.

First you need to install the latest Kinect toolkit and SDK from Microsoft. As many namespaces have been changed from the previous ones.

So to initialize a device follow these steps.

Create a new project, Can be a WPF or Windows Forms Project.

On the main page.

Add these two imports

using Microsoft.Kinect;

using Microsoft.Kinect.Toolkit;

Now we want to see how many Kinect devices are connected to the project or and there status. For that we have to make an object of type KinectSensorChooser that is inside the Kinect.Toolkit dll.

private readonly KinectSensorChooser sensorChooser = new KinectSensorChooser();

In the constructor, we can add this line.

sensorChooser.KinectChanged += sensorChooser_KinectChanged;  

sensorChooser.Start();

Above two lines will initialize the search for kinect devices, and as soon as it finds it, it will raise the KinectChanged event.

void sensorChooser_KinectChanged(object sender, KinectChangedEventArgs e)        

{            

           KinectSensor oldSensor = e.OldSensor;            

            StopKinect(oldSensor);

            KinectSensor newSensor = e.NewSensor;            

try           

  {                

     newSensor.Start();            

}            

catch (System.IO.IOException)            

{                

         sensorChooser.TryResolveConflict();

}        

}

This will initialize the newSensor object if a kinect device is found.