# US 2012/0268364 A1 · Fast fingertip detection for initializing a vision-based hand tracker

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2012/0268364 A1
- Document ID: US20120268364A1
- Status: Published application
- Kind: A1
- Application number: 13/430,509
- Filed: 2012-03-26
- Published: 2012-10-25
- Earliest claimed: 2008-04-24
- Assignees: Oblong Industries, Inc.
- Inventors: David MINNEN
- Web page: https://patentgrep.com/patent/US20120268364A1

## CPC

- G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)}
- G06F 3/0304: {Detection arrangements using opto-electronic means (constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042)}
- G06F 3/0325: {using a plurality of light emitters or reflectors or a plurality of detectors forming a reference frame from which to derive the orientation of the object, e.g. by triangulation or on the basis of reference deformation in the picked up image}
- G06V 40/107: {Static hand or arm}
- G06V 10/245: {by locating a pattern; Special marks for positioning}
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

Systems and methods for initializing real-time, vision-based hand tracking systems are described. The systems and methods for initializing the vision-based hand tracking systems image a body and receive gesture data that is absolute three-space data of an instantaneous state of the body at a point in time and space, and at least one of determine an orientation of the body using an appendage of the body and track the body using at least one of the orientation and the gesture data.

## Claims

### Claim 1

A method comprising:
receiving via an optical detector gesture data that is absolute three-space data of an instantaneous state of a body at a point in time and space; and
at least one of generating an estimated orientation of the body using an appendage of the body and tracking the body using at least one of the estimated orientation and the gesture data.

### Claim 2 (depends on 1)

The method of claim 1, comprising detecting the appendage of the body and identifying a segment of the appendage using the gesture data.

### Claim 3 (depends on 2)

The method of claim 2, comprising detecting the appendage of the body by discriminating between at least one first pixel region that includes the appendage and at least one second pixel region from which the appendage is absent.

### Claim 4 (depends on 2)

The method of claim 2, wherein the optical detector comprises a video detector outputting a plurality of video frames comprising the gesture data.

### Claim 5 (depends on 4)

The method of claim 4, comprising converting each video frame into a binary image.

### Claim 6 (depends on 4)

The method of claim 4, comprising pre-processing each video frame of the plurality of video frames, the pre-processing comprising the detecting of the appendage and the identifying of the segment of the appendage.

### Claim 7 (depends on 6)

The method of claim 6, comprising generating an estimated orientation of the body using data of the segment.

### Claim 8 (depends on 4)

The method of claim 4, comprising comparing each video frame of the plurality of video frames to a background model.

### Claim 9 (depends on 8)

The method of claim 8, comprising generating the background model by computing an adapting running average of the plurality of video frames.

### Claim 10 (depends on 9)

The method of claim 9, comprising converting the background model into a background binary image.

### Claim 11 (depends on 8)

The method of claim 8, comprising calculating for each pixel of the video frame a pixel value difference between a first pixel value of the respective pixel of the video frame and a second pixel value of each respective corresponding pixel of the background model.

### Claim 12 (depends on 11)

The method of claim 11, comprising generating marked pixels by marking each pixel of the video frame for which the pixel value difference exceeds a threshold value.

### Claim 13 (depends on 12)

The method of claim 12, comprising generating a foreground image comprising the marked pixels.

### Claim 14 (depends on 13)

The method of claim 13, comprising extracting at least one feature from the gesture data, wherein the at least one feature comprises at least one pixel of the foreground image.

### Claim 15 (depends on 14)

The method of claim 14, wherein the detecting comprises applying the at least one feature to a plurality of filters in turn, and identifying the at least one feature as including the appendage when the at least one feature is accepted by all filters of the plurality of filters.

### Claim 16 (depends on 14)

The method of claim 14, wherein the at least one feature comprises at least one box, wherein the at least one box is specified using a plurality of coordinates within at least one query region of the gesture data.

### Claim 17 (depends on 16)

The method of claim 16, comprising specifying the at least one box using an upper-left coordinate and a lower-right coordinate of the at least one box.

### Claim 18 (depends on 16)

The method of claim 16, wherein the at least one box is characterized by a number of pixels of the at least one foreground image.

### Claim 19 (depends on 16)

The method of claim 16, wherein the at least one feature comprises at least one ring, wherein the at least one ring is an unfilled rectangle region within a box.

### Claim 20 (depends on 19)

The method of claim 19, wherein the detecting comprises applying the at least one feature to a plurality of filters in turn, and identifying the at least one feature as including the appendage when the at least one feature is accepted by all filters of the plurality of filters.

### Claim 21 (depends on 20)

The method of claim 20, wherein the appendage comprises a fingertip.

### Claim 22 (depends on 21)

The method of claim 21, wherein the body comprises a finger.

### Claim 23 (depends on 20)

The method of claim 20, wherein the body comprises a hand.

### Claim 24 (depends on 20)

The method of claim 20, wherein the body comprises an arm.

### Claim 25 (depends on 20)

The method of claim 20, comprising, for each detected instance of the appendage, generating a body prediction corresponding to the appendage.

### Claim 26 (depends on 25)

The method of claim 25, wherein the body prediction comprises a predicted position corresponding to the appendage.

### Claim 27 (depends on 25)

The method of claim 25, wherein the body prediction comprises the estimated orientation corresponding to the detected region.

### Claim 28 (depends on 25)

The method of claim 25, comprising generating a body prediction list and storing the body prediction in the body prediction list, wherein the body prediction list comprises a plurality of body predictions.

### Claim 29 (depends on 28)

The method of claim 28, comprising generating an edge image for the video frame.

### Claim 30 (depends on 29)

The method of claim 29, wherein the generating of the edge image is provided by an edge detector.

### Claim 31 (depends on 29)

The method of claim 29, comprising generating an edge orientation image for each edge pixel of the edge image.

### Claim 32 (depends on 31)

The method of claim 31, wherein the generating of the edge orientation image comprises calculating an angle implied by local partial derivatives.

### Claim 33 (depends on 31)

The method of claim 31, comprising, when tracking of the body is in progress, generating a predicted estimate for the body prediction using a previous estimate of the body prediction from at least one previous video frame.

### Claim 34 (depends on 33)

The method of claim 33, comprising, when tracking of a plurality of body hypotheses is in progress, generating for each body hypothesis a predicted estimate for the body prediction using a previous estimate of the body prediction from at least one previous video frame.

### Claim 35 (depends on 33)

The method of claim 33, comprising removing from the body prediction list duplicates of the body predictions corresponding to the video frame.

### Claim 36 (depends on 33)

The method of claim 33, comprising, for each predicted estimate, identifying a best parameter value.

### Claim 37 (depends on 36)

The method of claim 36, comprising identifying the best parameter value using particle swarm optimization.

### Claim 38 (depends on 36)

The method of claim 36, comprising identifying the best parameter value using a set of body models.

### Claim 39 (depends on 38)

The method of claim 38, wherein the set of body models comprises a set of hand models.

### Claim 40 (depends on 38)

The method of claim 38, comprising generating the set of hand models by creating a contour that matches a shape of a hand in each pose of a plurality of poses.

### Claim 41 (depends on 40)

The method of claim 40, comprising representing the contour using a quadratic b-spline.

### Claim 42 (depends on 38)

The method of claim 38, wherein the best parameter value comprises at least one parameter value of a body prediction that best matches a body model of the set of body models, wherein the body prediction corresponds to the gesture data.

### Claim 43 (depends on 42)

The method of claim 42, wherein the best parameter value comprises at least one of position, scale, and orientation.

### Claim 44 (depends on 42)

The method of claim 42, wherein the best parameter value comprises position.

### Claim 45 (depends on 42)

The method of claim 42, wherein the best parameter value comprises scale.

### Claim 46 (depends on 42)

The method of claim 42, wherein the best parameter value comprises orientation.

### Claim 47 (depends on 42)

The method of claim 42, comprising, for each parameter value, updating control points of a hand spline of the body model.

### Claim 48 (depends on 47)

The method of claim 47, wherein the updating comprises updating via at least one of translation, scaling, and rotation.

### Claim 49 (depends on 47)

The method of claim 47, wherein the updating comprises updating via translation.

### Claim 50 (depends on 47)

The method of claim 47, wherein the updating comprises updating via scaling.

### Claim 51 (depends on 47)

The method of claim 47, wherein the updating comprises updating via rotation.

### Claim 52 (depends on 47)

The method of claim 47, comprising sampling along the hand spline.

### Claim 53 (depends on 52)

The method of claim 52, comprising comparing local contour orientation of the hand spline to oriented edges of the edge orientation image of the video frame.

### Claim 54 (depends on 53)

The method of claim 53, comprising comparing by searching along a normal of the hand spline for a closest edge with a matching orientation to the body prediction.

### Claim 55 (depends on 53)

The method of claim 53, comprising updating the plurality of predicted estimates using the best parameter values.

### Claim 56 (depends on 55)

The method of claim 55, comprising reporting as a current body location a highest scoring predicted estimate when the highest scoring predicted estimate exceeds a threshold.

### Claim 57 (depends on 56)

The method of claim 56, comprising reporting absence of a body in a corresponding frame when the highest scoring predicted estimate when the highest scoring predicted estimates is less than the threshold.

### Claim 58 (depends on 4)

The method of claim 4, comprising detecting a gesture of the body using the gesture data.

### Claim 59 (depends on 58)

The method of claim 58, wherein the gesture comprises at least one of poses and motion of the body.

### Claim 60 (depends on 58)

The method of claim 58, comprising translating the gesture into a gesture signal using a gesture notation.

### Claim 61 (depends on 60)

The method of claim 60, comprising controlling a component coupled to a computer using the gesture signal.

### Claim 62 (depends on 61)

The method of claim 61, comprising identifying the gesture, wherein the identifying includes identifying a pose and an orientation of a portion of the body.

### Claim 63 (depends on 62)

The method of claim 62, wherein the detecting comprises generating three-dimensional space point data representing the gesture and labeling the space point data.

### Claim 64 (depends on 63)

The method of claim 63, wherein the translating includes translating the space point data into commands appropriate to a configuration of the computer.

### Claim 65 (depends on 61)

The method of claim 61, wherein the translating comprises translating information of the gesture to a gesture notation.

### Claim 66 (depends on 65)

The method of claim 65, wherein the gesture notation represents a gesture vocabulary, and the gesture signal comprises communications of the gesture vocabulary.

### Claim 67 (depends on 66)

The method of claim 66, wherein the gesture vocabulary represents in textual form instantaneous pose states of kinematic linkages of the body.

### Claim 68 (depends on 66)

The method of claim 66, wherein the gesture vocabulary represents in textual form an orientation of kinematic linkages of the body.

### Claim 69 (depends on 66)

The method of claim 66, wherein the gesture vocabulary represents in textual form a combination of orientations of kinematic linkages of the body.

### Claim 70 (depends on 66)

The method of claim 66, wherein the gesture vocabulary includes a string of characters that represent a state of kinematic linkages of the body.

### Claim 71 (depends on 70)

The method of claim 70, comprising assigning each position in the string to the appendage.

### Claim 72 (depends on 71)

The method of claim 71, comprising assigning characters of a plurality of characters to each of a plurality of positions of the appendage.

### Claim 73 (depends on 72)

The method of claim 72, wherein the plurality of positions are established relative to a coordinate origin.

### Claim 74 (depends on 73)

The method of claim 73, comprising establishing the coordinate origin using a position selected from a group consisting of an absolute position and orientation in space, a fixed position and orientation relative to the body irrespective of an overall position and heading of the body, and interactively in response to an action of the body.

### Claim 75 (depends on 72)

The method of claim 72, comprising assigning characters of the plurality of characters to each of a plurality of orientations of the appendage.

### Claim 76 (depends on 61)

The method of claim 61, wherein controlling the component comprises controlling a three-space object in six degrees of freedom simultaneously by mapping the gesture of the appendage to the three-space object.

### Claim 77 (depends on 61)

The method of claim 61, wherein controlling the component comprises controlling a three-space object through three translational degrees of freedom and three rotational degrees of freedom.

### Claim 78 (depends on 77)

The method of claim 77, wherein the three-space object is presented on a display device coupled to the computer.

### Claim 79 (depends on 77)

The method of claim 77, wherein the three-space object is coupled to the computer.

### Claim 80 (depends on 77)

The method of claim 77, wherein the detecting comprises detecting when an extrapolated position of the object intersects virtual space, wherein the virtual space comprises space depicted on a display device coupled to the computer.

### Claim 81 (depends on 80)

The method of claim 80, wherein controlling the component comprises controlling a virtual object in the virtual space when the extrapolated position intersects the virtual object.

### Claim 82 (depends on 58)

The method of claim 58, comprising controlling scaling of the detecting and controlling to generate coincidence between virtual space and physical space, wherein the virtual space comprises space depicted on a display device coupled to the computer, wherein the physical space comprises space inhabited by the body.

### Claim 83 (depends on 82)

The method of claim 82, comprising determining dimensions, orientations, and positions in the physical space of a display device coupled to the computer.

### Claim 84 (depends on 83)

The method of claim 83, comprising dynamically mapping the physical space in which the display device is located as a projection into the virtual space of at least one application coupled to the computer.

### Claim 85 (depends on 82)

The method of claim 82, comprising translating scale, angle, depth, and dimension between the virtual space and the physical space as appropriate to at least one application coupled to the computer.

### Claim 86 (depends on 82)

The method of claim 82, comprising controlling at least one virtual object in the virtual space in response to movement of at least one physical object in the physical space.

### Claim 87 (depends on 82)

The method of claim 82, comprising controlling rendering of graphics on the display device in response to position of the body in physical space relative to position of the display device.

### Claim 88

A method comprising:
receiving via an optical detector gesture data that is absolute three-space data of an instantaneous state of a body at a point in time and space;
detecting an appendage of the body and identifying a region of the appendage using the gesture data; and
at least one of generating an estimated orientation of the body using the appendage and tracking the body using at least one of the estimated orientation and the gesture data.

### Claim 89 (depends on 88)

The method of claim 88, wherein the optical detector comprises a video detector outputting a plurality of video frames comprising the gesture data.

### Claim 90 (depends on 89)

The method of claim 89, comprising detecting motion in the plurality of video frames.

### Claim 91 (depends on 90)

The method of claim 90, comprising detecting the appendage by discriminating between at least one first pixel region of a video frame that includes the appendage and at least one second pixel region of the video frame from which the appendage is absent.

### Claim 92 (depends on 91)

The method of claim 91, comprising detecting edge pixels of the video frame.

### Claim 93 (depends on 92)

The method of claim 92, comprising generating an edge image for the video frame.

### Claim 94 (depends on 93)

The method of claim 93, comprising generating an edge orientation image for each edge pixel of the edge image.

### Claim 95 (depends on 94)

The method of claim 94, wherein the generating of the edge orientation image comprises calculating an angle implied by local partial derivatives of the video frame.

### Claim 96 (depends on 94)

The method of claim 94, comprising generating a body prediction using tracking data when the tracking data is available.

### Claim 97 (depends on 96)

The method of claim 96, comprising generating a body prediction using at least one of the edge pixels, the edge orientation image, and the appendage.

### Claim 98 (depends on 97)

The method of claim 97, comprising detecting the body by comparing the body prediction to a set of body models.

### Claim 99 (depends on 98)

The method of claim 98, comprising tracking the body.

### Claim 100 (depends on 99)

The method of claim 99, comprising detecting a gesture of the body using the gesture data.

### Claim 101 (depends on 100)

The method of claim 100, wherein the gesture comprises at least one of poses and motion of the body.

### Claim 102 (depends on 100)

The method of claim 100, comprising translating the gesture into a gesture signal using a gesture notation.

### Claim 103 (depends on 102)

The method of claim 102, comprising controlling a computer application using the gesture signal.

### Claim 104

A method comprising:
receiving via an optical detector gesture data that is absolute three-space data of an instantaneous state of the body at a point in time and space;
detecting a segment of an appendage of the body and generating an estimated orientation of the body using the segment; and
tracking the body using at least one of estimated orientation and the gesture data.

### Claim 105

A method comprising:
receiving via an optical detector gesture data that is absolute three-space data of an instantaneous state of the body at a point in time and space;
detecting an appendage of the body and identifying a segment of the appendage using the gesture data;
generating an estimated orientation of the body using data of the segment; and
tracking the body using at least one of the estimated orientation and the gesture data.

### Claim 106

A method comprising:
detecting a gesture of a body from gesture data received via an optical detector, wherein the gesture data is absolute three-space location data of an instantaneous state of the body at a point in time and space, the detecting comprising detecting a segment of an appendage of the body and generating an estimated orientation of the body using the segment;
tracking the body using at least one of the estimated orientation and the gesture data and, while tracking, identifying the gesture using the gesture data; and
generating a gesture signal that represents the gesture and providing the gesture signal as a control signal.

### Claim 107

A method comprising:
receiving via an optical detector gesture data that is absolute three-space data of an instantaneous state of a body at a point in time and space;
at least one of generating an estimated orientation of the body using an appendage of the body and tracking the body using at least one of the estimated orientation and the gesture data;
detecting and identifying a gesture of the body using the gesture data, and translating the gesture to a gesture signal; and
controlling a component coupled to a computer in response to the gesture signal.

### Claim 108

A method comprising:
receiving via an optical detector gesture data that is absolute three-space data of an instantaneous state of the body at a point in time and space;
detecting an appendage of the body and identifying a segment of the appendage using the gesture data, and determining an orientation of the body using the segment;
tracking the body using at least one of the orientation and the gesture data;
detecting and identifying a gesture of the body using only the gesture data;
translating the gesture to a gesture signal; and
controlling a computer component using the gesture signal.
