# US 12,223,674 B2 · Methods and systems for joint pose and shape estimation of objects from sensor data

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,223,674 B2
- Document ID: US12223674B2
- Status: Granted
- Kind: B2
- Application number: 18/318,771
- Filed: 2023-05-17
- Published: 2023-09-14
- Granted: 2025-02-11
- Earliest claimed: 2019-11-15
- Assignees: Volkswagen Group of America Investments, LLC
- Applicants: Argo AI, LLC
- Inventors: Hunter Goforth; Xiaoyan Hu; Michael Happold
- Web page: https://patentgrep.com/patent/US12223674B2

## CPC

- G06T 7/73: using feature-based methods
- B60W 30/09: Taking automatic action to avoid collision, e.g. braking and steering
- B60W 30/0956: {the prediction being responsive to traffic or environmental parameters}
- B60W 60/0016: {of the vehicle or its occupants}
- B60W 60/0027: {using trajectory prediction for other traffic participants}
- B60W 60/00272: {relying on extrapolation of current movement}
- G06N 3/045: Combinations of networks
- G06N 3/0455: Auto-encoder networks; Encoder-decoder networks
- G06N 3/0499: Feedforward networks
- G06N 3/08: Learning methods
- G06N 3/088: Non-supervised learning, e.g. competitive learning
- G06N 3/09: Supervised learning
- G06T 7/50: Depth or shape recovery
- G06T 7/60: Analysis of geometric attributes
- G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80)
- G06V 10/242: {by image rotation, e.g. by 90 degrees}
- G06V 10/255: {Detecting or recognising potential candidate objects based on visual cues, e.g. shapes}
- G06V 10/761: {Proximity, similarity or dissimilarity measures}
- G06V 10/82: using neural networks
- G06V 20/58: Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
- G06V 20/64: Three-dimensional [3D] objects
- B60W 2420/403: Image sensing, e.g. optical camera
- B60W 2420/408: Radar; Laser, e.g. lidar
- B60W 2554/404: Characteristics
- G06N 3/047: Probabilistic or stochastic networks
- G06T 2207/10028: Range image; Depth image; 3D point clouds
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/30244: Camera pose
- G06T 2207/30261: Obstacle
- G06V 2201/08: Detecting or categorising vehicles
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- B60W: CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

Methods and systems for jointly estimating a pose and a shape of an object perceived by an autonomous vehicle are described. The system includes data and program code collectively defining a neural network which has been trained to jointly estimate a pose and a shape of a plurality of objects from incomplete point cloud data. The neural network includes a trained shared encoder neural network, a trained pose decoder neural network, and a trained shape decoder neural network. The method includes receiving an incomplete point cloud representation of an object, inputting the point cloud data into the trained shared encoder, outputting a code representative of the point cloud data. The method also includes generating an estimated pose and shape of the object based on the code. The pose includes at least a heading or a translation and the shape includes a denser point cloud representation of the object.

## Claims

### Claim 1

A method comprising, by a processor:
receiving point cloud data corresponding to an object, wherein the point cloud data is an incomplete point cloud representation of the object;
accessing a non-transitory computer readable medium storing data defining a neural network which has been trained to jointly estimate a pose and a shape of the object of a plurality of objects from incomplete point cloud data, the neural network comprising a trained shared encoder neural network, a trained pose decoder neural network, and a trained shape decoder neural network, wherein the neural network was trained by freezing or predetermining one or more parameters associated with the trained shared encoder neural network and then using corresponding codes generated by the trained shared encoder neural network to train a pose decoder neural network;
outputting, by the trained shared encoder neural network, a code representative of the point cloud data that can be used for estimating both the pose and the shape of the object;
generating, by the trained pose decoder neural network, an estimated pose of the object based on the code, the estimated pose comprising at least a heading or a translation of the object; and
generating, by the trained shape decoder neural network, an estimated shape of the object based on the code, the estimated shape comprising an estimated point cloud representation of the object that is denser than the incomplete point cloud representation of the object.

### Claim 2 (depends on 1)

The method of claim 1, wherein:
the point cloud data is collected by an autonomous vehicle during navigation; and
the method further comprises:
using the estimated shape and the estimated pose of the object to forecast a trajectory of the object in an environment of the autonomous vehicle; and
causing, based on the forecasted trajectory, the autonomous vehicle to take an action to avoid colliding with the object.

### Claim 3 (depends on 1)

The method of claim 1, further comprising using a LIDAR sensor to generate the point cloud data.

### Claim 4 (depends on 1)

The method of claim 1, wherein the code is a lower dimensional representation of the point could data comprising a feature vector derived from the point cloud data.

### Claim 5 (depends on 1)

The method of claim 1, wherein the trained shared encoder neural network comprises at least two deep network layers, each of the at least two deep network layers comprising a multilayer perceptron.

### Claim 6

A method, by a processor:
receiving point cloud data corresponding to an object, wherein the point cloud data is an incomplete point cloud representation of the object;
accessing a non-transitory computer readable medium storing data defining a neural network which has been trained to jointly estimate a pose and a shape of the object of a plurality of objects from incomplete point cloud data, the neural network comprising a trained shared encoder neural network, a trained pose decoder neural network, and a trained shape decoder neural network;
outputting, by the trained shared encoder neural network, a code representative of the point cloud data that can be used for estimating both the pose and the shape of the object;
generating, by the trained pose decoder neural network, an estimated pose of the object based on the code, the estimated pose comprising at least a heading or a translation of the object; and
generating, by the trained shape decoder neural network, an estimated shape of the object based on the code, the estimated shape comprising an estimated point cloud representation of the object that is denser than the incomplete point cloud representation of the object;
wherein the neural network is trained to jointly estimating the pose and the shape of the object of the plurality of objects by:
receiving training data comprising partial point cloud data and ground truth data corresponding to one or more objects;
generating the trained shared encoder and the trained shape decoder neural network by using the training data to collectively train a shared encoder neural network and a shape decoder neural network, wherein the trained shared encoder neural network and the trained shared decoder neural network are trained to estimate the shape of the object of the plurality of objects;
generating the trained pose decoder neural network by using one or more codes generated by the trained shared encoder neural network to train a pose decoder neural network while freezing one or more parameters associated with the trained shared encoder neural network; and
outputting the neural network comprising the trained shared encoder neural network, the trained pose decoder neural network, and the trained shape decoder neural network.

### Claim 7 (depends on 6)

The method of claim 6, wherein generating the trained shared encoder neural network and the trained shape decoder neural network further comprises freezing one or more parameters associated with the pose decoder neural network.

### Claim 8 (depends on 6)

The method of claim 6, further comprising using a loss function to constrain training of the shared encoder neural network and the shape decoder neural network, wherein the loss function minimizes a distance between estimated shape completion and ground truth shape completion.

### Claim 9 (depends on 8)

The method of claim 8, wherein the loss function comprises a Chamfer Distance loss function.

### Claim 10 (depends on 6)

The method of claim 6, further comprising using a pose loss function to constrain training of the pose decoder neural network, wherein the pose loss function minimizes a distance between estimated point pose and ground truth pose.

### Claim 11 (depends on 6)

The method of claim 6, further comprising fine-tuning the trained neural network by collectively training the trained shared encoder neural network, the trained shape decoder neural network, and the trained pose decoder neural network to generate an updated trained neural network, the collective training constrained by a joint loss function that includes a shape loss function and a pose loss function.

### Claim 12 (depends on 6)

The method of claim 6, wherein the training data comprises synthetic data generated using simulation.

### Claim 13

A system, comprising:
a processor; and
a non-transitory computer readable medium comprising:
data and program code collectively defining a neural network which has been trained to jointly estimate a pose and a shape of an object of a plurality of objects from incomplete point cloud data, the neural network comprising a trained shared encoder neural network, a trained pose decoder neural network, and a trained shape decoder neural network, wherein the neural network was trained by freezing or predetermining one or more parameters associated with the trained shared encoder neural network and then using corresponding codes generated by the trained shared encoder neural network to train a pose decoder neural network, and
one or more programming instructions that, when executed by the processor, cause the processor to:
receive point cloud data corresponding to the object, wherein the point cloud data is an incomplete point cloud representation of the object,
outputting, by the trained shared encoder neural network, a code representative of the point cloud data that can be used for estimating both the pose and the shape of the object,
generating, by the trained pose decoder neural network, an estimated pose of the object based on the code, the estimated pose comprising at least a heading or a translation of the object, and
generating, by the trained shape decoder neural network, an estimated shape of the object based on the code, the estimated shape comprising an estimated point cloud representation of the object that is denser than the incomplete point cloud representation of the object.

### Claim 14 (depends on 13)

The system of claim 13, wherein:
the point cloud data is collected by an autonomous vehicle during navigation; and
the system further comprises programming instructions that, when executed by the processor, cause the processor to:
use the estimated shape and the estimated pose of the object to forecast a trajectory of the object in an environment of the autonomous vehicle, and
cause, based on the forecasted trajectory, the autonomous vehicle to take an action to avoid colliding with the object.

### Claim 15 (depends on 13)

The system of claim 13, further comprising programming instructions that, when executed by the processor, cause the processor to use a LIDAR sensor to generate the point cloud data.

### Claim 16 (depends on 13)

The system of claim 13, wherein the trained shared encoder neural network comprises at least two deep network layers, each of the at least two deep network layers comprising a multilayer perceptron.

### Claim 17

A system, comprising:
a processor; and
a non-transitory computer readable medium comprising:
data and program code collectively defining a neural network which has been trained to jointly estimate a pose and a shape of an object of a plurality of objects from incomplete point cloud data, the neural network comprising a trained shared encoder neural network, a trained pose decoder neural network, and a trained shape decoder neural network, and
one or more programming instructions that when executed by the processor, cause the processor to:
receive point cloud data corresponding to the object, wherein the point cloud data is an incomplete point cloud representation of the object,
outputting, by the trained shared encoder neural network, a code representative of the point cloud data that can be used for estimating both the pose and the shape of the object,
generating, by the trained pose decoder neural network, an estimated pose of the object based on the code, the estimated pose comprising at least a heading or a translation of the object, and
generating, by the trained shape decoder neural network, an estimated shape of the object based on the code, the estimated shape comprising an estimated point cloud representation of the object that is denser than the incomplete point cloud representation of the object,
wherein the neural network is trained to jointly estimate the pose and the shape of the object by:
receiving training data comprising partial point cloud data and ground truth data corresponding to one or more objects;
generating the trained shared encoder neural network and the trained shape decoder neural network by using the training data to collectively train a shared encoder neural network and a shape decoder neural network, wherein the trained shared encoder neural network and the trained shared decoder neural network are trained to estimate the shape of the object of the plurality of objects;
generating a trained pose decoder by using one or more codes generated by the trained shared encoder neural network to train a pose decoder neural network while freezing one or more parameters associated with the trained shared encoder neural network; and
outputting the neural network comprising the trained shared encoder neural network, the trained pose decoder neural network, and the trained shape decoder neural network.

### Claim 18 (depends on 17)

The system of claim 17, wherein generating the trained shared encoder neural network and the trained shape decoder neural network further comprises freezing one or more parameters associated with the pose decoder neural network.

### Claim 19 (depends on 17)

The system of claim 17, further comprising programming instructions that, when executed by the processor, cause the processor to generate the neural network trained to jointly estimating the pose and the shape of the object of the plurality of objects by using a loss function to constrain training of the shared encoder neural network and the shape decoder neural network, wherein the loss function minimizes a distance between estimated shape completion and ground truth shape completion.

### Claim 20 (depends on 19)

The system of claim 19, further comprising programming instructions that, when executed by the processor, cause the processor to generate the neural network trained to jointly estimating the pose and the shape of the object of the plurality of objects by using a pose loss function to constrain training of the pose decoder neural network, wherein the pose loss function minimizes a distance between estimated point pose and ground truth pose.