Sign based human-machine interaction

A computer based method for man-machine interaction comprising the steps of:

• receiving a digital image of a target, the digital image including depth data associated with each pixel and being indicative of the distance to a camera capturing the digital image;
• pre-processing the digital image to determine a location of at least a hand and at least one of a head or a torso of the target based on the depth data associated with each pixel;
• matching a shape of the hand with one of a plurality of stored hand shape images to provide a matched shape associated with the digital image, wherein matching includes forming a vector with the depth data associated with each of the pixels associated with the location of the hand and computing a match based on a matrix of projection coefficient vectors representing the plurality of stored hand shape images;
• creating a candidate image data object including information associated with the location of a hand with respect to one of a head or a torso of the target and information indicating the matched shape associated with the digital image;
• matching a gesture captured by one or more digital images with a template gesture by comparing the candidate image data object with one ore more stored gesture profiles for template gestures, the gesture profiles including hand to body part location information and hand shape information for comparing with the candidate image data object;
• and
• matching a candidate shape of the hand by generating a skeleton representation of the hand from the image pixel data of the location of the hand and determining one or more hand attributes from the skeleton representation, including at least a number of fingers.

A computer based method for human-machine interaction comprising:

• matching a hand shape captured in one or more digital images with a plurality of hand shape patterns to determine a candidate gesture hand shape, each of the one or more digital images including depth data associated with each pixel indicative of the distance to a camera capturing the digital images, wherein matching includes forming a vector with the depth data associated with each of the pixels and computing a match based on a matrix of projection coefficient vectors representing the plurality of stored hand shape patterns;
• determining a trajectory curve of the hand from the one ore more digital images;
• matching the trajectory curve of the hand with a plurality of trajectory curve templates to determine a candidate gesture motion;
• determining a gesture corresponding to the candidate gesture hand shape and the candidate gesture motion by comparing the candidate gesture hand shape and the candidate gesture motion with a plurality of gesture profiles with associated gesture hand shapes and gesture motions;
• and
• matching a candidate shape of the hand by generating a skeleton representation of the hand from the image pixel data of the location of the hand and determining one or more hand attributes from the skeleton representation, including at least a number of fingers.

A system for human-machine interaction comprising:

• an image data pre-processing module for receiving frames of image data comprising human image information, each of the frames of image data including depth data associated with each pixel indicative of the distance to a camera that captured the frame, the image pre-processing module configured to determine locations of one or more body parts of the human in the frames based on the depth data associated with each pixel;
• a shape matching module coupled to the image data pre-processing module for receiving the determined locations and configured to match a shape of at least one of the body parts by comparing information associated with the image data with stored shape profiles in a shape database, wherein comparing includes forming a vector with the depth data associated with each of the pixels associated with a determined location of one or more body parts and computing a match based on a matrix of projection coefficient vectors representing the plurality of stored body part shape images;
• and
• a gesture matching module coupled to the image data-preprocessing module and the shape matching module for receiving shape and location information, the gesture matching module configured to match a gesture of the at least one body part by comparing gesture attributes of the at least one body part with gesture profiles stored in a gesture database, the gesture attributes comprising values associated with body part locations, and shape information, the gesture matching module further comprising a hand shape matching module configured to match a candidate shape of the hand by generating a skeleton representation of the hand from the image pixel data of the location of the hand and to determine one or more hand attributes from the skeleton representation, including at least a number of fingers.

A computer readable media for human-machine interaction comprising:

• program instructions for receiving a digital image of a target, the digital image including depth data associated with each pixel indicative of the distance to a camera that captured the digital image;
• program instructions for pre-processing the digital image to determine a location of at least a hand and at least one of a head or a torso of the target based on the depth data associated with each pixel;
• program instructions for matching a shape of the hand with one of a plurality of stored hand shape images to provide a matched shape associated with the digital image, wherein the program instructions for matching include program instructions for forming a vector with the depth data associated with each of the pixels associated with the location of the hand and for computing a match based on a matrix of projection coefficient vectors representing the plurality of stored hand shape images;
• program instructions for creating a candidate image data object including information associated with the location of a hand with respect to one of a head or a torso of the target and information indicating the matched shape associated with the digital image;
• program instructions for matching a gesture captured by one or more digital images with a template gesture by comparing the candidate image data object with one ore more stored gesture profiles for template gestures, the gesture profiles including hand to body part location information and hand shape information for comparing with the candidate image data object;
• and
• program instructions for matching a candidate shape of the hand by generating a skeleton representation of the hand from the image pixel data of the location of the hand and determining one or more hand attributes from the skeleton representation, including at least a number of fingers.

A computer based system for human-machine interaction comprising:

• means for receiving a digital image of a target, the digital image including depth data associated with each pixel indicative of the distance to a camera that captured the digital image;
• means for pre-processing the digital image to determine a location of at least a hand and at least one of a head or a torso of the target based on the depth data associated with each pixel;
• means for matching a shape of the hand with one of a plurality of stored hand shape images to provide a matched shape associated with the digital image, wherein the means for matching include means for forming a vector with the depth data associated with each of the pixels associated with the location of the hand and means for computing a match based on a matrix of projection coefficient vectors representing the plurality of stored hand shape images;
• means for creating a candidate image data object including information associated with the location of a hand with respect to one of a head or a torso of the target and information indicating the matched shape associated with the digital image;
• and
• means for matching a gesture captured by one or more digital images with a template gesture by comparing the candidate image data object with one ore more stored gesture profiles for template gestures, the gesture profiles including hand to body part location information and hand shape information for comparing with the candidate image data object;
• and
• means for matching a candidate shape of the hand by generating a skeleton representation of the hand from the image pixel data of the location of the hand and determining one or more hand attributes from the skeleton representation, including at least a number of fingers.

A computer based method for man-machine interaction comprising the steps of:

• receiving a digital image of a target having a hand, the digital image comprising depth data associated with image pixels, the depth data corresponding to the distance from the target to a camera capturing the digital image;
• determining a location of a hand of the target within the pixels of the image based on the depth data associated with each of the pixels;
• and
• generating a skeleton representation of the hand from the image pixel data of the location of the hand, wherein generating includes determining characteristics of a first skeleton representation of the hand and applying a rule to the determined characteristics to delete at least one line segment in the first skeleton representation resulting in a second skeleton representation, wherein generating includes:
• determining an end point and an intersection point for a line in the first skeleton representation of the hand;
• determining the distance from the end point to the intersection point of the line;
• comparing the distance to a distance threshold characterizing a minimum distance for a finger;
• and deleting the line in the first skeleton representation based on the comparison to the distance threshold thereby removing a false finger skeleton line and resulting in a second skeleton representation.