# US 10,372,228 B2 · Method and system for 3D hand skeleton tracking

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,372,228 B2
- Document ID: US10372228B2
- Status: Granted
- Kind: B2
- Application number: 15/649,716
- Filed: 2017-07-14
- Published: 2018-01-25
- Granted: 2019-08-06
- Earliest claimed: 2016-07-20
- Assignees: Usens, Inc.
- Applicants: uSens, Inc.
- Inventors: Wentao Mao; Xu Zhang; Gengyu Ma; Yue Fei
- Web page: https://patentgrep.com/patent/US10372228B2

## CPC

- G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)}
- A63F 13/213: comprising photodetecting means, e.g. cameras, photodiodes or infrared cells (A63F13/219 takes precedence)
- A63F 13/428: involving motion or position input signals, e.g. signals representing the rotation of an input controller or a player's arm motions sensed by accelerometers or gyroscopes
- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06T 7/251: {involving models}
- G06T 7/285: using a sequence of stereo image pairs
- G06T 7/75: {involving models}
- G06V 40/113: {Recognition of static hand signs}
- G06V 40/28: {Recognition of hand or arm movements, e.g. recognition of deaf sign language (static hand signs G06V40/113)}
- A63F 2300/6045: for mapping control signals received from the input arrangement into game commands
- G06T 2207/30196: Human being; Person
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- A63F: CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

A tracking system is disclosed. The system may comprise a processor and a non-transitory computer-readable storage medium coupled to the processor and storing instructions that, when executed by the processor, cause the system to perform a method. The method may comprise training a detection model and an extraction model, capturing one or more images of at least a portion of an object, detecting the portion of the object in each of the one or more images through the trained detection model, tracking the detected portion of the object in real-time, obtaining 2D positions of one or more locations on the tracked portion of the object through the trained extraction model, and obtaining 3D positions of the one or more locations on the tracked portion of the object based at least in part on the obtained 2D positions.

## Claims

### Claim 1

A tracking system, comprising:
a processor; and
a non-transitory computer-readable storage medium coupled to the processor and storing instructions that, when executed by the processor, cause the system to perform:
training a detection model and an extraction model, comprising:
obtaining images of various hands of different people in different postures,
identifying hand positions in the obtained images according to verified hand data as a ground truth of hand,
training a first machine learning model with the obtained images and the ground truth of hand to obtain the detection model,
using the detection model to predict hand positions in the obtained images, and
training a second machine learning model with the ground truth of hand and the predicted hand positions in the obtained images to refine the detection model;
capturing one or more images of at least a portion of an object, wherein the portion of the object comprises a hand;
detecting the portion of the object in each of the one or more images through the trained detection model;
tracking the detected portion of the object in real-time;
obtaining 2D positions of one or more locations on the tracked portion of the object through the trained extraction model, wherein the one or more locations comprise one or more joints of the hand; and
obtaining 3D positions of the one or more locations on the tracked portion of the object based at least in part on the obtained 2D positions.

### Claim 2 (depends on 1)

The tracking system of claim 1, wherein:
the one or more images comprise two stereo images of the portion of the object; and
the system further comprises two infrared cameras configured to capture the two stereo images.

### Claim 3 (depends on 1)

The tracking system of claim 1, wherein:
the first machine learning model includes at least one of a random forest method or a convolution neural network (CNN) method.

### Claim 4 (depends on 1)

The tracking system of claim 1, wherein:
the second machine learning model includes at least one of a random forest method or a convolution neural network (CNN) method.

### Claim 5 (depends on 1)

The tracking system of claim 1, wherein:
the hand positions are positions of bounding boxes defining the hand; and
detecting the portion of the object in each of the one or more images through the trained detection model comprises detecting the portion of the object in each of the one or more images through the detection model trained from at least one of the first or the second machine learning model.

### Claim 6 (depends on 5)

The tracking system of claim 5, wherein training the detection model and the extraction model further comprises:
identifying positions of hand joints in the obtained images according to verified joint data as a ground truth of joints;
using the refined detection model to obtain cropped regions of the hand, the cropped regions corresponding to the bounding boxes; and
training a third machine learning model with the cropped regions of the hand and the ground truth of joints to obtain the extraction model, the third machine learning model including at least one of a random forest method or a convolution neural network (CNN) method.

### Claim 7 (depends on 6)

The tracking system of claim 6, wherein training the detection model and the extraction model further comprises:
using the extraction model to predict positions of joints of the hand; and
training a fourth machine learning model with the predicted positions of the joints and the ground truth of joints to refine the extraction model, the fourth machine learning model including at least one of a random forest method or a convolution neural network (CNN) method.

### Claim 8 (depends on 7)

The tracking system of claim 7, wherein obtaining the 2D positions of the one or more locations on the tracked portion of the object through the trained extraction model comprises obtaining the 2D positions of the one or more locations on the tracked portion of the object through the extraction model trained from at least one of the third or the fourth machine learning model.

### Claim 9 (depends on 6)

The tracking system of claim 6, further comprising applying at least one of Kalman Filter, extended Kalman Filter, Mean Filter, or Medium Filter.

### Claim 10 (depends on 1)

The tracking system of claim 1, wherein tracking the detected portion of the object in real-time comprises detecting the portion of the object continuously based on at least one of Particle Swarm Optimization (PSO), Particle Filter, Kalman Filter, extended Kalman Filter, Markov Chain Method, or Monte Carlo Method.

### Claim 11 (depends on 1)

The tracking system of claim 1, wherein obtaining the 3D positions of the one or more locations on the tracked portion of the object based at least in part on the obtained 2D positions comprises obtaining the 3D positions of the one or more locations on the tracked portion of the object based at least in part on:
pairs of 2D joint positions from the two stereo images;
focal lengths of cameras respectively capturing the two stereo images; and
position information of the cameras.

### Claim 12 (depends on 1)

The tracking system of claim 1, further comprising subjecting the obtained 3D positions of the one or more locations to one or more constraints to obtain refined 3D positions of the one or more locations, wherein the one or more constraints comprise a normal range of each hand bone length corresponding to distances among the hand joints.

### Claim 13 (depends on 12)

The tracking system of claim 12, further comprising at least one a first and a second optimization methods, wherein:
the first optimization method comprises:
projecting the refined 3D positions to 2D to obtain projected 2D positions, comparing the projected 2D positions with the obtained 2D positions to obtain a first difference, and adjusting the refined 3D positions to minimize the first difference, obtaining optimized 3D positions; and
the second optimization method comprises:
comparing the refined 3D positions with the obtained 3D positions to obtain a second difference, and adjusting the refined 3D positions to minimize the second difference, obtaining optimized 3D positions.

### Claim 14 (depends on 1)

The tracking system of claim 1, wherein:
the tracking system is disposed on a mobile device;
the mobile device is coupled to at least one of a local device or a cloud device;
the tracking system is configured to send failed detections to at least one of the local device or the cloud device;
the local device is configured to correct the failed detection, further training at least one of the detection or the extraction model; and
the cloud device is configured to correct the failed detection, further training at least one of the detection or the extraction model.

### Claim 15

A tracking method, comprising:
training a detection model and an extraction model, comprising:
obtaining images of various hands of different people in different postures,
identifying hand positions in the obtained images according to verified hand data as a ground truth of hand,
training a first machine learning model with the obtained images and the ground truth of hand to obtain the detection model,
using the detection model to predict hand positions in the obtained images, and
training a second machine learning model with the ground truth of hand and the predicted hand positions in the obtained images to refine the detection model;
capturing one or more images of at least a portion of an object, wherein the portion of the object comprises a hand;
detecting the portion of the object in each of the one or more images through the trained detection model;
tracking the detected portion of the object in real-time;
obtaining 2D positions of one or more locations on the tracked portion of the object through the trained extraction model, wherein the one or more locations comprise one or more joints of the hand; and
obtaining 3D positions of the one or more locations on the tracked portion of the object based at least in part on the obtained 2D positions.

### Claim 16 (depends on 15)

The tracking method of claim 15, wherein:
the one or more images comprise two stereo images of the portion of the object; and
the system comprises two infrared cameras configured to capture the two stereo images.

### Claim 17 (depends on 15)

The tracking method of claim 15, wherein:
the first machine learning model includes at least one of a random forest method or a convolution neural network (CNN) method.

### Claim 18 (depends on 15)

The tracking method of claim 15, wherein:
the second machine learning model includes at least one of a random forest method or a convolution neural network (CNN) method.

### Claim 19 (depends on 15)

The tracking method of claim 15, wherein:
the hand positions are positions of bounding boxes defining the hand; and
detecting the portion of the object in each of the one or more images through the trained detection model comprises detecting the portion of the object in each of the one or more images through the detection model trained from at least one of the first or the second machine learning model.

### Claim 20 (depends on 15)

The tracking method of claim 15, wherein tracking the detected portion of the object in real-time comprises detecting the portion of the object continuously based on at least one of Particle Swarm Optimization (PSO), Particle Filter, Kalman Filter, extended Kalman Filter, Markov Chain Method, or Monte Carlo Method.

### Claim 21 (depends on 19)

The tracking method of claim 19, wherein training the detection model and the extraction model further comprises:
identifying positions of hand joints in the obtained images according to verified joint data as a ground truth of joints;
using the refined detection model to obtain cropped regions of the hand, the cropped regions corresponding to the bounding boxes; and
training a third machine learning model with the cropped regions of the hand and the ground truth of joints to obtain the extraction model, the third machine learning model including at least one of a random forest method or a convolution neural network (CNN) method.

### Claim 22 (depends on 21)

The tracking method of claim 21, further comprising applying at least one of Kalman Filter, extended Kalman Filter, Mean Filter, or Medium Filter.

### Claim 23 (depends on 21)

The tracking method of claim 21, wherein training the detection model and the extraction model further comprises:
using the extraction model to predict positions of joints of the hand; and
training a fourth machine learning model with the predicted positions of the joints and the ground truth of joints to refine the extraction model, the fourth machine learning model including at least one of a random forest method or a convolution neural network (CNN) method.

### Claim 24 (depends on 23)

The tracking method of claim 23, wherein obtaining the 2D positions of the one or more locations on the tracked portion of the object through the trained extraction model comprises obtaining the 2D positions of the one or more locations on the tracked portion of the object through the extraction model trained from at least one of the third or the fourth machine learning model.

### Claim 25 (depends on 15)

The tracking method of claim 15, wherein obtaining the 3D positions of the one or more locations on the tracked portion of the object based at least in part on the obtained 2D positions comprises obtaining the 3D positions of the one or more locations on the tracked portion of the object based at least in part on:
pairs of 2D joint positions from the two stereo images;
focal lengths of cameras respectively capturing the two stereo images; and
position information of the cameras.

### Claim 26 (depends on 15)

The tracking method of claim 15, further comprising subjecting the obtained 3D positions of the one or more locations to one or more constraints to obtain refined 3D positions of the one or more locations, wherein the one or more constraints comprise a normal range of each hand bone length corresponding to distances among the hand joints.

### Claim 27 (depends on 26)

The tracking method of claim 26, further comprising at least one a first and a second optimization methods, wherein:
the first optimization method comprises:
projecting the refined 3D positions to 2D to obtain projected 2D positions, comparing the projected 2D positions with the obtained 2D positions to obtain a first difference, and adjusting the refined 3D positions to minimize the first difference, obtaining optimized 3D positions; and
the second optimization method comprises:
comparing the refined 3D positions with the obtained 3D positions to obtain a second difference, and adjusting the refined 3D positions to minimize the second difference, obtaining optimized 3D positions.

### Claim 28 (depends on 15)

The tracking method of claim 15, wherein:
the tracking system is disposed on a mobile device;
the mobile device is coupled to at least one of a local device or a cloud device;
the tracking system is configured to send failed detections to at least one of the local device or the cloud device;
the local device is configured to correct the failed detection, further training at least one of the detection or the extraction model; and
the cloud device is configured to correct the failed detection, further training at least one of the detection or the extraction model.

### Claim 29

A non-transitory computer-readable storage medium storing instructions that, when executed by a processor, perform:
training a detection model and an extraction model, comprising:
obtaining images of various hands of different people in different postures,
identifying hand positions in the obtained images according to verified hand data as a ground truth of hand,
training a first machine learning model with the obtained images and the ground truth of hand to obtain the detection model,
using the detection model to predict hand positions in the obtained images, and
training a second machine learning model with the ground truth of hand and the predicted hand positions in the obtained images to refine the detection model;
capturing one or more images of at least a portion of an object, wherein the portion of the object comprises a hand;
detecting the portion of the object in each of the one or more images through the trained detection model;
tracking the detected portion of the object in real-time;
obtaining 2D positions of one or more locations on the tracked portion of the object through the trained extraction model, wherein the one or more locations comprise one or more joints of the hand; and
obtaining 3D positions of the one or more locations on the tracked portion of the object based at least in part on the obtained 2D positions.

### Claim 30 (depends on 29)

The medium of claim 29, wherein obtaining the 3D positions of the one or more locations on the tracked portion of the object based at least in part on the obtained 2D positions comprises obtaining the 3D positions of the one or more locations on the tracked portion of the object based at least in part on:
pairs of 2D joint positions from the two stereo images;
focal lengths of cameras respectively capturing the two stereo images; and
position information of the cameras.