# US 11,908,155 B2 · Efficient pose estimation through iterative refinement

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,908,155 B2
- Document ID: US11908155B2
- Status: Granted
- Kind: B2
- Application number: 17/203,607
- Filed: 2021-03-16
- Published: 2022-09-22
- Granted: 2024-02-20
- Earliest claimed: 2021-03-16
- Assignees: QUALCOMM Incorporated
- Applicants: QUALCOMM Incorporated
- Inventors: John Yang; Yash Sanjay Bhalgat; Fatih Murat Porikli; Simyung Chang
- Web page: https://patentgrep.com/patent/US11908155B2

## CPC

- G06N 3/063: using electronic means
- G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80)
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/08: Learning methods
- G06N 3/09: Supervised learning
- G06N 3/092: Reinforcement learning
- G06N 20/00: Machine learning
- G06V 40/11: {Hand-related biometrics; Hand pose recognition}
- G06T 2207/20081: Training; Learning
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

Certain aspects of the present disclosure provide a method, including: processing input data with a feature extraction stage of a machine learning model to generate a feature map; applying an attention map to the feature map to generate an augmented feature map; processing the augmented feature map with a refinement stage of the machine learning model to generate a refined feature map; processing the refined feature map with a first regression stage of the machine learning model to generate multi-dimensional task output data; and processing the refined feature data with an attention stage of the machine learning model to generate an updated attention map.

## Claims

### Claim 1

A method, comprising:
processing input data with a feature extraction stage of a machine learning model to generate a feature map;
applying an attention map to the feature map to generate an augmented feature map;
selecting a batch normalization layer, from a plurality of batch normalization layers, based on a loop count;
processing the augmented feature map with a refinement stage of the machine learning model to generate a refined feature map, wherein processing the augmented feature map with the refinement stage of the machine learning model comprises applying the batch normalization layer to the augmented feature map;
processing the refined feature map with a first regression stage of the machine learning model to generate multi-dimensional task output data; and
processing the refined feature map with an attention stage of the machine learning model to generate an updated attention map.

### Claim 2 (depends on 1)

The method of claim 1, further comprising processing the multi-dimensional task output data with a three-dimensional position generator to generate a three-dimensional position estimate.

### Claim 3 (depends on 2)

The method of claim 2, further comprising processing the three-dimensional position estimate with the multi-dimensional task output data to generate a two-dimensional position estimate.

### Claim 4 (depends on 3)

The method of claim 3, wherein the three-dimensional position generator comprises a body element pose model.

### Claim 5 (depends on 4)

The method of claim 4, wherein:
the three-dimensional position estimate comprises one or more body element joint positions in a three-dimensional space,
the two-dimensional position estimate comprises the one or more body element joint positions in a two-dimensional space, and
the multi-dimensional task output data comprises one or more of:
a rotation estimate,
a translation estimate, or
a scaling estimate.

### Claim 6 (depends on 5)

The method of claim 5, wherein the body element pose model comprises a hand model.

### Claim 7 (depends on 5)

The method of claim 5, further comprising processing the refined feature map with a second regression stage of the machine learning model to generate a task uncertainty estimate.

### Claim 8 (depends on 7)

The method of claim 7, further comprising:
determining an uncertainty associated with the three-dimensional position estimate based on the task uncertainty estimate; and
determining an uncertainty associated with the two-dimensional position estimate based on the task uncertainty estimate.

### Claim 9 (depends on 7)

The method of claim 7, further comprising:
processing the task uncertainty estimate with a loop decision stage of the machine learning model to generate a loop decision value; and
determining whether to exit processing of the machine learning model based on the loop decision value.

### Claim 10 (depends on 9)

The method of claim 9, further comprising determining to exit processing of the machine learning model based on the loop decision value.

### Claim 11 (depends on 9)

The method of claim 9, further comprising:
determining not to exit processing of the machine learning model based on the loop decision value; and
determining to exit processing of the machine learning model based on a loop count.

### Claim 12 (depends on 5)

The method of claim 5, further comprising displaying the two-dimensional position estimate on a display screen of an extended reality device.

### Claim 13 (depends on 1)

The method of claim 1, further comprising:
determining a training loss based on a training loss function, wherein the training loss function comprises:
an uncertainty loss component; and
a task loss component; and
updating the machine learning model based on the training loss.

### Claim 14

A processing system, comprising:
one or more processors individually or collectively configured to execute instructions stored on one or more memories and to cause the processing system to:
process input data with a feature extraction stage of a machine learning model to generate a feature map;
apply an attention map to the feature map to generate an augmented feature map;
select a batch normalization layer from a plurality of batch normalization layers based on a loop count;
process the augmented feature map with a refinement stage of the machine learning model to generate a refined feature map, wherein in order to process the augmented feature map with the refinement stage of the machine learning model, the one or more processors are further configured to cause the processing system to apply the batch normalization layer to the augmented feature map;
process the refined feature map with a first regression stage of the machine learning model to generate multi-dimensional task output data; and
process the refined feature map with an attention stage of the machine learning model to generate an updated attention map.

### Claim 15 (depends on 14)

The processing system of claim 14, wherein the one or more processors are further configured to cause the processing system to process the multi-dimensional task output data with a three-dimensional position generator to generate a three-dimensional position estimate.

### Claim 16 (depends on 15)

The processing system of claim 15, wherein the one or more processors are further configured to cause the processing system to process the three-dimensional position estimate with the multi-dimensional task output data to generate a two-dimensional position estimate.

### Claim 17 (depends on 16)

The processing system of claim 16, wherein the three-dimensional position generator comprises a body element pose model.

### Claim 18 (depends on 17)

The processing system of claim 17, wherein:
the three-dimensional position estimate comprises one or more body element joint positions in a three-dimensional space,
the two-dimensional position estimate comprises the one or more body element joint positions in a two-dimensional space, and
the multi-dimensional task output data comprises one or more of:
a rotation estimate,
a translation estimate, or
a scaling estimate.

### Claim 19 (depends on 18)

The processing system of claim 18, wherein the body element pose model comprises a hand model.

### Claim 20 (depends on 18)

The processing system of claim 18, wherein the one or more processors are further configured to cause the processing system to display the two-dimensional position estimate on a display screen of an extended reality device.

### Claim 21 (depends on 16)

The processing system of claim 16, wherein the one or more processors are further configured to cause the processing system to process the refined feature map with a second regression stage of the machine learning model to generate a task uncertainty estimate.

### Claim 22 (depends on 21)

The processing system of claim 21, wherein the one or more processors are further configured to cause the processing system to:
determine an uncertainty associated with the three-dimensional position estimate based on the task uncertainty estimate; and
determine an uncertainty associated with the two-dimensional position estimate based on the task uncertainty estimate.

### Claim 23 (depends on 21)

The processing system of claim 21, wherein the one or more processors are further configured to cause the processing system to:
process the task uncertainty estimate with a loop decision stage of the machine learning model to generate a loop decision value; and
determine whether to exit processing of the machine learning model based on the loop decision value.

### Claim 24 (depends on 23)

The processing system of claim 23, wherein the one or more processors are further configured to cause the processing system to determine to exit processing of the machine learning model based on the loop decision value.

### Claim 25 (depends on 23)

The processing system of claim 23, wherein the one or more processors are further configured to cause the processing system to:
determine not to exit processing of the machine learning model based on the loop decision value; and
determine to exit processing of the machine learning model based on a loop count.

### Claim 26 (depends on 14)

The processing system of claim 14, wherein the one or more processors are further configured to cause the processing system to:
determine a training loss based on a training loss function, wherein the training loss function comprises:
an uncertainty loss component; and
a task loss component; and
update the machine learning model based on the training loss.

### Claim 27

A non-transitory computer-readable medium comprising computer-executable instructions that, when executed by one or more processors of a processing system, cause the processing system to perform a method, the method comprising:
processing input data with a feature extraction stage of a machine learning model to generate a feature map;
applying an attention map to the feature map to generate an augmented feature map;
selecting a batch normalization layer, from a plurality of batch normalization layers, based on a loop count;
processing the augmented feature map with a refinement stage of the machine learning model to generate a refined feature map, wherein processing the augmented feature map with the refinement stage of the machine learning model comprises applying the batch normalization layer to the augmented feature map;
processing the refined feature map with a first regression stage of the machine learning model to generate multi-dimensional task output data; and
processing the refined feature map with an attention stage of the machine learning model to generate an updated attention map.

### Claim 28

A processing system, comprising:
means for processing input data with a feature extraction stage of a machine learning model to generate a feature map;
means for applying an attention map to the feature map to generate an augmented feature map;
means for selecting a batch normalization layer, from a plurality of batch normalization layers, based on a loop count;
means for processing the augmented feature map with a refinement stage of the machine learning model to generate a refined feature map, wherein means for processing the augmented feature map with the refinement stage of the machine learning model comprise means for applying the batch normalization layer to the augmented feature map;
means for processing the refined feature map with a first regression stage of the machine learning model to generate multi-dimensional task output data; and
means for processing the refined feature map with an attention stage of the machine learning model to generate an updated attention map.