# US 12,008,787 B2 · Object pose estimation

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,008,787 B2
- Document ID: US12008787B2
- Status: Granted
- Kind: B2
- Application number: 17/380,174
- Filed: 2021-07-20
- Published: 2023-01-26
- Granted: 2024-06-11
- Earliest claimed: 2021-07-20
- Assignees: Ford Global Technologies, LLC
- Applicants: Ford Global Technologies, LLC
- Inventors: Shubham Shrivastava; Gaurav Pandey; Punarjay Chakravarty
- Web page: https://patentgrep.com/patent/US12008787B2

## CPC

- G06T 7/74: {involving reference images or patches}
- B60W 10/04: including control of propulsion units
- B60W 10/18: including control of braking systems
- B60W 10/20: including control of steering systems
- G05B 13/027: {using neural networks only}
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06N 3/0895: Weakly supervised learning, e.g. semi-supervised or self-supervised learning
- G06T 7/75: {involving models}
- G06T 2207/10024: Color image
- G06T 2207/10028: Range image; Depth image; 3D point clouds
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/30204: Marker
- G06T 2207/30232: Surveillance
- G06T 2207/30236: Traffic on road, railway or crossing
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- B60W: CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
- G05B: CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS (systems for controlling or regulating non-electric variables G05D; systems for regulating electric or magnetic variables G05F; control devices or systems insofar as characterised by mechanical features only G05G)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

A depth image of an object can be input to a deep neural network to determine a first four degree-of-freedom pose of the object. The first four degree-of-freedom pose and a three-dimensional model of the object can be input to a silhouette rendering program to determine a first two-dimensional silhouette of the object. A second two-dimensional silhouette of the object can be determined based on thresholding the depth image. A loss function can be determined based on comparing the first two-dimensional silhouette of the object to the second two-dimensional silhouette of the object. Deep neural network parameters can be optimized based on the loss function and the deep neural network can be output.

## Claims

### Claim 1

A computer, comprising:
a processor; and
a memory, the memory including instructions executable by the processor to:
input a depth image of an object to a deep neural network to determine a first four degree-of-freedom pose of the object;
input the first four degree-of-freedom pose and a three-dimensional model of the object to a silhouette rendering program to determine a first two-dimensional silhouette of the object;
threshold the depth image to determine a second two-dimensional silhouette of the object;
determine a loss function based on comparing the first two-dimensional silhouette of the object which is based on the first four degree-of-freedom pose and the three-dimensional model of the object to the second two-dimensional silhouette of the object which is based on the depth image of the object;
optimize deep neural network parameters based on the loss function; and
output the deep neural network.

### Claim 2 (depends on 1)

The computer of claim 1, wherein the first four degree-of-freedom pose is determined based on x, y, and z coordinates and an angle in an x, y plane.

### Claim 3 (depends on 1)

The computer of claim 1, the instructions including further instructions to determine a translation matrix and a rotation matrix based on the first four degree-of-freedom pose.

### Claim 4 (depends on 3)

The computer of claim 3, wherein the translation matrix and the rotation matrix permit differentiation with respect to the first four degree-of-freedom pose.

### Claim 5 (depends on 1)

The computer of claim 1, the instructions including further instructions to determine the loss function by determining a distance between the first two-dimensional silhouette of the object to the second two-dimensional silhouette of the object.

### Claim 6 (depends on 5)

The computer of claim 5, wherein the distance is one or more of an L1 distance, a chamfer distance, and a centroid distance.

### Claim 7 (depends on 1)

The computer of claim 1, the instructions including further instructions to optimize deep neural network parameters based on the loss function by backpropagating the loss function through the deep neural network.

### Claim 8 (depends on 1)

The computer of claim 1, wherein the deep neural network is a convolutional neural network that includes convolutional layers and fully connected layers.

### Claim 9 (depends on 1)

The computer of claim 1, wherein the three-dimensional model of the object is determined based on one or more of computer aided design data or a lidar scan.

### Claim 10 (depends on 1)

The computer of claim 1, the instructions including further instructions to determine a second four degree-of-freedom pose of the object based on a red, green, blue image and fiducial markers applied to the object.

### Claim 11 (depends on 10)

The computer of claim 10, the instructions including further instructions to determine a second loss function based on comparing the second four degree-of-freedom pose of the object with the first four degree-of-freedom pose of the object.

### Claim 12 (depends on 1)

The computer of claim 1, the instructions including further instructions to determine a vehicle path based determining on one or more four degree-of-freedom poses for a vehicle.

### Claim 13 (depends on 12)

The computer of claim 12, the instructions including further instructions to operate the vehicle on the vehicle path by controlling one or more of vehicle powertrain, vehicle brakes, and vehicle steering.

### Claim 14

A method, comprising:
inputting a depth image of an object to a deep neural network to determine a first four degree-of-freedom pose of the object;
inputting the first four degree-of-freedom pose and a three-dimensional model of the object to a silhouette rendering program to determine a first two-dimensional silhouette of the object;
thresholding the depth image to determine a second two-dimensional silhouette of the object;
determining a loss function based on comparing the first two-dimensional silhouette of the object which is based on the first four degree-of-freedom pose and the three-dimensional model of the object to the second two-dimensional silhouette of the object which is based on the depth image of the object;
optimizing deep neural network parameters based on the loss function; and
outputting the deep neural network.

### Claim 15 (depends on 14)

The method of claim 14, wherein the first four degree-of-freedom pose is determined based on x, y, and z coordinates and an angle in an x, y plane.

### Claim 16 (depends on 14)

The method of claim 14, further comprising determining a translation matrix and a rotation matrix based on the first four degree-of-freedom pose.

### Claim 17 (depends on 16)

The method of claim 16, wherein the translation matrix and the rotation matrix permit differentiation with respect to the first four degree-of-freedom pose.

### Claim 18 (depends on 14)

The method of claim 14, further comprising determining the loss function by determining a distance between the first two-dimensional silhouette of the object to the second two-dimensional silhouette of the object.

### Claim 19 (depends on 18)

The method of claim 18, wherein the distance is one or more of an L1 distance, a chamfer distance, and a centroid distance.

### Claim 20 (depends on 14)

The method of claim 14, further comprising optimizing deep neural network parameters based on the loss function by backpropagating the loss function through the deep neural network.