US 12,749,214 B2Granted

Three-dimensional pose estimation using two-dimensional images

AssigneeNVIDIA Corporation

InventorsSakthivel Sivaraman, Arjun Guru, Rajath Shetty

Life of this patent

Estimating the term…

Loading patent…

Abstract

In various examples, systems and methods for pose detection model training for predicting three-dimensional pose estimates using two-dimensional image data are provided. The occupant pose detection model may be trained using multi-view image sensor training data that includes image frames that capture a pose of a training subject within a machine interior using multiple synchronized optical image sensors placed around the machine interior that produce a set of captured image frames of the training subject from different viewpoints. Based on the multi-view image sensor training data, the occupant pose detection model may generate a set of individual, predicted 3D pose estimates for the training subject from a captured image frame from each of the respective optical image sensors. To adjust the occupant pose detection model during training, a loss feedback may be generated that comprises a pose alignment loss, a pose depth loss, and/or a ground truth kinematic loss.

Claims18

Dependent claims

A system comprising:

one or more processing units to:
apply to at least one pose detection model, multi-view image sensor training data comprising two-dimensional optical image data representing an occupant of a machine, the two-dimensional optical image data captured from a plurality of viewpoints by a plurality of synchronized optical image sensors;
input reference depth data representing the occupant derived from a depth sensor synchronized with the plurality of synchronized optical image sensors;
generate a plurality of three-dimensional pose estimates of the occupant based on the two-dimensional optical image data captured from the plurality of viewpoints;
generate a loss feedback comprising at least an alignment loss representing an alignment between the plurality of three-dimensional pose estimates for one or more kinematic elements of the occupant and a pose depth loss representing a difference between a depth of the one or more kinematic elements represented by the plurality of three-dimensional pose estimates and a depth of the one or more kinematic elements represented by the reference depth data;
and
adjust the at least one pose detection model based on the loss feedback.

4 more elements

At least one processor comprising:

one or more processing units to:
operate at least one pose detection model to generate a plurality of three-dimensional pose estimates of an occupant based on two-dimensional optical image data captured from a plurality of viewpoints;
input reference depth data representing the occupant derived from a depth sensor;
generate a loss feedback comprising at least an alignment loss representing an alignment between the plurality of three-dimensional pose estimates and a pose depth loss representing a difference between a depth of one or more kinematic elements represented by the plurality of three-dimensional pose estimates and a depth of the one or more kinematic elements represented by the reference depth data;
and
adjust the at least one pose detection model based on the loss feedback.

3 more elements

Dependencies are inferred from the claim text.

Facts

Application: 18/349,842Filed 10 Jul 2023
Published: US 2025/0022155 A116 Jan 2025
Granted: 29 Sep 2026
Term ends: Not estimatedThe source records don’t support an estimate.

Classification7 codes

G06T 7/70
Image analysis: determining position or orientation of objects or cameras
G06T 7/50
Image analysis: depth or shape recovery
G06T 7/60
Image analysis: analysis of geometric attributes
G06T 7/80
Image analysis: analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration