Gesture recognition for wearable multimedia device using real-time data streams

A system comprising:

• at least one processor;
• memory storing instructions that when executed by the at least one processor, cause the at least on processor to perform operations comprising:
• detecting a trigger event from two or more real-time data streams running on a wearable multimedia device, each of the two or more real-time data streams carrying sensor data from a sensor of the wearable multimedia device, wherein the two or more real-time data streams include a first data stream carrying a depth image frame from a depth sensor and a second data stream carrying a two-dimensional (2D) image frame from a camera;
• inferring that a user wearing or holding the wearable multimedia device is pointing to or holding an object, the inferring comprising:
• projecting each pixel in the depth image frame into three-dimensional (3D) space using camera intrinsic parameters;
• generating a point cloud of 3D data points from the projected pixels;
• reducing a number of 3D data points in the point cloud to make the 3D data points more spatially uniform;
• dividing the reduced number of 3D points into potential hand clusters;
• deleting clusters with too few points;
• identifying for each potential hand cluster if it is a left or right hand by checking from which side arm points enter the depth image frame;
• removing the arm points from the identified hand cluster;
• determining a dominant axis of the hand cluster that represents the hand direction;
• isolating a finger cluster from the hand cluster;
• determining a dominant axis of the finger cluster to determine a finger pointing direction;
• projecting a 2D bounding box for the hand into 2D camera image frame using the camera intrinsic parameters;
• generating a list of candidate held objects by intersecting the 2D bounding box for the hand with one or more labeled 2D bounding boxes containing one or more recognized objects;
• for each candidate held object, determining a center point of the 2D bounding box containing the object and a distance from the center point to the hand cluster;
• and
• selecting the closest candidate held object as the held object based on the distances.