# US 2024/0199068 A1 · Object pose estimation

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2024/0199068 A1
- Document ID: US20240199068A1
- Status: Published application
- Kind: A1
- Application number: 18/244,050
- Filed: 2023-09-08
- Published: 2024-06-20
- Earliest claimed: 2022-11-18
- Assignees: NVIDIA Corporation
- Applicants: NVIDIA Corporation
- Inventors: Marco Pavone; Heng Yang
- Web page: https://patentgrep.com/patent/US20240199068A1

## CPC

- B60W 60/001: {Planning or execution of driving tasks}
- B60W 50/0097: {Predicting future conditions}
- G06T 7/73: using feature-based methods
- B60W 2420/403: Image sensing, e.g. optical camera
- G06T 2200/24: involving graphical user interfaces [GUIs]
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/20092: Interactive image processing based on input by user
- G06T 2207/30252: Vehicle exterior; Vicinity of vehicle
- B60W: CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

Apparatuses, systems, and techniques to obtain prediction set(s) (e.g., region(s)) for keypoint prediction(s) based at least in part on data associated with an object, compute a set of candidate poses for the object based at least in part on the prediction set(s), and estimate an estimated object pose based at least in part on the set of candidate poses. The estimated object pose may be used to move a device. For example the estimated object pose may be used to provide collision-free motion generation for a real-world or virtual device (e.g., a robot, an autonomous machine, or a semi-autonomous machine). In at least one embodiment, at least a portion of the object pose estimation and/or at least a portion of the collision-free motion generation is performed in parallel.

## Claims

### Claim 1

An autonomous machine configured to:
receive an estimated object pose; and
move based at least in part on the estimated object pose,
wherein the estimated object pose is created by at least one computing system, by,
obtaining at least one region for at least one keypoint prediction generated based at least in part on an image depicting an object,
generating a set of candidate poses for the object based at least in part on the at least one region, and
generating the estimated object pose based at least in part on the set of candidate poses.

### Claim 2 (depends on 1)

The autonomous machine of claim 1, further comprising:
the at least one computing device.

### Claim 3 (depends on 1)

The autonomous machine of claim 1, further comprising:
at least one image capture device to capture the image.

### Claim 4 (depends on 1)

The autonomous machine of claim 1, further comprising:
a user interface to receive a probability from a user, wherein the at least one region has at least the probability of comprising at least one ground truth keypoint, and the set of candidate poses has at least the probability of including a ground truth object pose.

### Claim 5 (depends on 1)

The autonomous machine of claim 1, wherein generating the estimated object pose comprises averaging at least a portion of the set of candidate poses.

### Claim 6

A method comprising:
obtaining, using at least one computing system, at least one region for at least one keypoint prediction generated based at least in part on an image depicting an object;
generating, using the at least one computing system, a set of candidate poses for the object based at least in part on the at least one region; and
estimating, using the at least one computing system, an estimated object pose based at least in part on the set of candidate poses.

### Claim 7 (depends on 6)

The method of claim 6, further comprising:
causing an autonomous machine or a semi-autonomous machine to move based at least in part on the estimated object pose.

### Claim 8 (depends on 6)

The method of claim 6, wherein obtaining the at least one region comprises conformalizing at least one heatmap generated based at least in part on the image.

### Claim 9 (depends on 8)

The method of claim 8, further comprising:
generating the at least one keypoint prediction and the at least one heatmap based at least in part on the image.

### Claim 10 (depends on 6)

The method of claim 6, wherein the at least one region comprises at least one of a circular region or an elliptical region.

### Claim 11 (depends on 6)

The method of claim 6, wherein inductive conformal prediction (“ICP”) is used to obtain the at least one region.

### Claim 12 (depends on 6)

The method of claim 6, wherein the set of candidate poses comprise a Pose UnceRtainty SEt (“PURSE”) having a first probability of including a ground truth object pose that is equal to a second probability of the at least one region comprising at least one ground truth keypoint.

### Claim 13 (depends on 12)

The method of claim 12, wherein the second probability is to be provided by a user.

### Claim 14 (depends on 6)

The method of claim 6, wherein estimating the estimated object pose comprises averaging at least a portion of the set of candidate poses.

### Claim 15 (depends on 6)

The method of claim 6, wherein estimating the estimated object pose comprises performing a convex optimization based at least in part on the set of candidate poses.

### Claim 16 (depends on 6)

The method of claim 6, further comprising:
determining an upper bound on at least one of a worst-case rotation error or a worst case translation error between the estimated object pose and a ground truth pose.

### Claim 17

A system comprising:
at least one processor; and
memory storing instructions that when executed by the at least one processor cause the system to:
obtain at least one region for at least one keypoint prediction generated based at least in part on an image depicting an object;
compute a set of candidate poses for the object based at least in part on the at least one region; and
estimate an estimated object pose based at least in part on the set of candidate poses.

### Claim 18 (depends on 17)

The system of claim 17, further comprising:
a movable device, the instructions, when executed by the at least one processor, to cause the system to cause the movable device to move based at least in part on the estimated object pose.

### Claim 19 (depends on 17)

The system of claim 17, further comprising:
at least one image capture device to capture the image.

### Claim 20 (depends on 17)

The system of claim 17, wherein the at least one region is obtained using at least one heatmap generated based at least in part on the image.

### Claim 21 (depends on 20)

The system of claim 20, wherein the instructions, when executed by the at least one processor, cause the system to compute the at least one keypoint prediction and the at least one heatmap based at least in part on the image.

### Claim 22 (depends on 17)

The system of claim 17, wherein the at least one region comprises at least one of a circular region or an elliptical region.

### Claim 23 (depends on 17)

The system of claim 17, wherein inductive conformal prediction (“ICP”) is used to obtain the at least one region.

### Claim 24 (depends on 17)

The system of claim 17, wherein the set of candidate poses comprise a Pose UnceRtainty SEt (“PURSE”) having a first probability of including a ground truth object pose that is equal to a second probability of the at least one region comprising at least one ground truth keypoint.

### Claim 25 (depends on 24)

The system of claim 24, further comprising:
a user interface, wherein the instructions, when executed by the at least one processor, cause the system to receive the second probability from the user interface.

### Claim 26 (depends on 17)

The system of claim 17, wherein estimating the estimated object pose comprises averaging at least a portion of the set of candidate poses.

### Claim 27 (depends on 17)

The system of claim 17, wherein the instructions, when executed by the at least one processor, cause the system to determine an upper bound on at least one of a worst-case rotation error or a worst case translation error between the estimated object pose and a ground truth pose.

### Claim 28

At least one processor comprising:
one or more circuits to obtain at least one prediction set for at least one keypoint prediction based at least in part on data associated with an object, compute a set of candidate poses for the object based at least in part on the at least one prediction set, and estimate an estimated object pose based at least in part on the set of candidate poses.

### Claim 29 (depends on 28)

The at least one processor of claim 28, wherein the one or more circuits are to cause a movable device to move based at least in part on the estimated object pose.

### Claim 30 (depends on 28)

The at least one processor of claim 28, wherein at least one heatmap is used to obtain the at least one prediction set.

### Claim 31 (depends on 28)

The at least one processor of claim 28, wherein the one or more circuits are to compute the at least one keypoint prediction based at least in part on an image.

### Claim 32 (depends on 28)

The at least one processor of claim 28, wherein the at least one prediction set comprises locations within at least one of a circular region or an elliptical region.

### Claim 33 (depends on 28)

The at least one processor of claim 28, wherein inductive conformal prediction (“ICP”) is used to obtain the at least one prediction set.

### Claim 34 (depends on 28)

The at least one processor of claim 28, wherein estimating the estimated object pose comprises averaging at least a portion of the set of candidate poses.

### Claim 35 (depends on 28)

The at least one processor of claim 28, wherein the set of candidate poses comprise a Pose UnceRtainty SEt (“PURSE”) having a first probability of including a ground truth object pose that is equal to a second probability of the at least one prediction set comprising at least one ground truth keypoint.

### Claim 36 (depends on 28)

The at least one processor of claim 28, wherein the one or more circuits are to determine an upper bound on at least one of a worst-case rotation error or a worst case translation error between the estimated object pose and a ground truth pose.

### Claim 37 (depends on 36)

The at least one processor of claim 36, wherein the upper bound is determined using one or more semi-definite relaxation processes.

### Claim 38 (depends on 28)

The at least one processor of claim 28, wherein the one or more circuits are to obtain the at least one prediction set by conformalizing the at least one keypoint prediction.

### Claim 39 (depends on 28)

The at least one processor of claim 28, wherein the one or more circuits are to obtain the at least one prediction set by conformalizing at least one heatmap generated based at least in part on an image.
