# US 12,008,822 B2 · Using neural networks for 3D surface structure estimation based on real-world data for autonomous systems and applications

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,008,822 B2
- Document ID: US12008822B2
- Status: Granted
- Kind: B2
- Application number: 17/452,752
- Filed: 2021-10-28
- Published: 2023-05-04
- Granted: 2024-06-11
- Earliest claimed: 2021-10-28
- Applicants: NVIDIA Corporation
- Inventors: Kang Wang; Yue Wu; Minwoo Park; Gang Pan
- Web page: https://patentgrep.com/patent/US12008822B2

## CPC

- G06V 20/64: Three-dimensional [3D] objects
- G01S 17/89: for mapping or imaging
- G01S 17/931: of land vehicles
- G06F 18/214: Generating training patterns; Bootstrap methods, e.g. bagging or boosting
- G06V 20/58: Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
- B60G 17/0165: to an external condition, e.g. rough road surface, side wind
- B60K 31/00: Vehicle fittings, acting on a single sub-unit only, for automatically controlling vehicle speed, i.e. preventing speed from exceeding an arbitrarily established velocity or maintaining speed at a particular velocity, as selected by the vehicle operator
- B60W 60/001: {Planning or execution of driving tasks}
- B60W 2420/408: Radar; Laser, e.g. lidar
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G01S: RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- B60G: VEHICLE SUSPENSION ARRANGEMENTS (air-cushion vehicles B60V; {cycle suspensions B62K25/00})
- B60K: ARRANGEMENT OR MOUNTING OF PROPULSION UNITS OR OF TRANSMISSIONS IN VEHICLES; ARRANGEMENT OR MOUNTING OF PLURAL DIVERSE PRIME-MOVERS IN VEHICLES; AUXILIARY DRIVES FOR VEHICLES; INSTRUMENTATION OR DASHBOARDS FOR VEHICLES; ARRANGEMENTS IN CONNECTION WITH COOLING, AIR INTAKE, GAS EXHAUST OR FUEL SUPPLY OF PROPULSION UNITS IN VEHICLES
- B60W: CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT

## Abstract

In various examples, to support training a deep neural network (DNN) to predict a dense representation of a 3D surface structure of interest, a training dataset is generated from real-world data. For example, one or more vehicles may collect image data and LiDAR data while navigating through a real-world environment. To generate input training data, 3D surface structure estimation may be performed on captured image data to generate a sparse representation of a 3D surface structure of interest (e.g., a 3D road surface). To generate corresponding ground truth training data, captured LiDAR data may be smoothed, subject to outlier removal, subject to triangulation to filling missing values, accumulated from multiple LiDAR sensors, aligned with corresponding frames of image data, and/or annotated to identify 3D points on the 3D surface of interest, and the identified 3D points may be projected to generate a dense representation of the 3D surface structure.

## Claims

### Claim 1

A method comprising:
generating, based at least on image data captured during a capture session in an environment, a first representation of a three-dimensional (3D) surface structure of a component of the environment;
generating, based at least on LiDAR data associated with the image data and captured during the capture session, a second representation of the 3D surface structure of the component; and
training one or more neural networks (NNs) to generate a densified representation of the 3D surface Lure using the first representation of the 3D surface structure as input training data and using the second representation of the 3D surface structure as around truth training data.

### Claim 2 (depends on 1)

The method of claim 1, wherein the generating of the first representation of the 3D surface structure comprises applying 3D surface structure estimation to the image data.

### Claim 3 (depends on 1)

The method of claim 1, wherein the generating of the second representation of the 3D surface structure comprises:
identifying an aligned set of the image data and the LiDAR data; and
generating the second representation of the 3D surface structure based at least on the aligned set of the image data and the LiDAR data.

### Claim 4 (depends on 1)

The method of claim 1, wherein the generating of the second representation of the 3D surface structure comprises filling in one or more missing values of the LiDAR data using Delaunay triangulation.

### Claim 5 (depends on 1)

The method of claim 1, wherein the generating of the second representation of the 3D surface structure comprises accumulating the LiDAR data from multiple LiDAR sensors.

### Claim 6 (depends on 1)

The method of claim 1, wherein the generating of the second representation of the 3D surface structure comprises:
by a labeling tool, obtaining one or more inputs identifying annotated points of the LiDAR data that correspond to the component of the environment, while presenting an aligned set of the image data and the LiDAR data; and
generating the second representation of the 3D surface structure of the component based at least on projecting the annotated points to form a projection image.

### Claim 7 (depends on 1)

The method of claim 1, further comprising applying object detection to the image data to generate a ground truth segmentation mask demarcating the component of the environment.

### Claim 8 (depends on 1)

The method of claim 1, wherein the second representation of the 3D surface structure of the component comprises a two-dimensional (2D) ground truth height map representing one or more height values of the component, wherein the training of the one or more NNs comprises comparing a predicted 2D height map with the 2D ground truth height map and applying a ground truth segmentation mask demarcating the component to cancel out one or more updates to the one or more NNs based on one or more values of the predicted 2D height map that do not correspond to the component.

### Claim 9 (depends on 1)

The method of claim 1, wherein the one or more NNs include a first output channel that regresses one or more height values of the component and a second output channel that regresses one or more confidence values corresponding to the one or more height values, wherein the training of the one or more NNs comprises applying a loss function, wherein the loss function compares predicted and ground truth height values and is based on predicted confidence values.

### Claim 10 (depends on 1)

The method of claim 1, further comprising using the one or more NNs to perform at least one of:
adapting a suspension system of an ego-object,
navigating the ego-object to avoid a detected protuberance, or
applying an acceleration or deceleration to the ego-object based at least on a detected surface slope.

### Claim 11

A processor comprising one or more circuits to:
access image data and LiDAR data captured during a capture session in an environment;
generate, based at least on the image data, an incomplete representation of a three-dimensional (3D) surface structure of a road in the environment;
generate, based at least on labeling of the LiDAR data, a second representation of the 3D surface structure of the road; and
train one or more neural networks (NNs) to generate a densified representation of the 3D surface structure using the incomplete representation of the 3D surface structure as input training data and using the second representation of the 3D surface structure as ground truth training data.

### Claim 12 (depends on 11)

The processor of claim 11, the one or more circuits further to generate the incomplete representation of the 3D surface structure based at least on applying 3D surface structure estimation to the image data.

### Claim 13 (depends on 11)

The processor of claim 11, the one or more circuits further to generate the second representation of the 3D surface structure based at least on filling in one or more missing values of the LiDAR data using Delaunay triangulation.

### Claim 14 (depends on 11)

The processor of claim 11, the one or more circuits further to generate the second representation of the 3D surface structure based at least on accumulating the LiDAR data from multiple LiDAR sensors.

### Claim 15 (depends on 11)

The processor of claim 11, the one or more circuits further to generate the second representation of the 3D surface structure based at least on:
by a labeling tool, obtaining one or more inputs identifying one or more annotated points of the LiDAR data that correspond to the road, while presenting an aligned set of the image data and the LiDAR data; and
generating the second representation of the 3D surface structure of the road based at least on projecting the one or more annotated points to form a projection image.

### Claim 16 (depends on 11)

The processor of claim 11, wherein the second representation of the 3D surface structure of the road comprises a two-dimensional (2D) ground truth height map representing one or more height values of the road, the one or more circuits further to train the one or more NNs based at least on comparing a predicted 2D height map with the 2D ground truth height map and applying a ground truth segmentation mask demarcating the road to cancel out one or more updates to the one or more NNs based on one or more values of the predicted 2D height map that do not correspond to the road.

### Claim 17 (depends on 11)

The processor of claim 11, the one or more NNs include a first output channel that regresses one or more height values of the road and a second output channel that regresses one or more confidence values corresponding to the one or more height values, the one or more circuits further to train the one or more NNs based at least on applying a loss function, wherein the loss function compares predicted and ground truth height values and is based on predicted confidence values.

### Claim 18

A system comprising:
one or more processing units; and
one or more memory units storing instructions that, when executed by the one or more processing units, cause the one or more processing units to execute operations comprising:
capturing image data and LiDAR data during a capture session in an environment;
generating, based at least on applying 3D structure estimation to the image data, an incomplete representation of a three-dimensional (3D) surface structure of a surface in the environment;
generating, based at least on labeling of the LiDAR data, a ground truth representation of the 3D surface structure of the surface; and
including the incomplete representation and the ground truth representation of the 3D surface structure in a training dataset.

### Claim 19 (depends on 18)

The system of claim 18, wherein the generating of the incomplete representation of the 3D surface structure comprises applying 3D surface structure estimation to the image data.

### Claim 20 (depends on 18)

The system of claim 18, wherein the generating of the ground truth representation of the 3D surface structure comprises:
by a labeling tool, obtaining one or more inputs identifying one or more annotated points of the LiDAR data that correspond to the surface in the environment, while presenting an aligned set of the image data and the LiDAR data; and
generating the ground truth representation of the 3D surface structure of the surface based at least on projecting the one or more annotated points to form a projection image.

### Claim 21 (depends on 18)

The system of claim 18, wherein the system is comprised of at least one of:
a control system for an autonomous or semi-autonomous machine;
a perception system for an autonomous or semi-autonomous machine;
a system for performing simulation operations;
a system for performing deep learning operations;
a system implemented using an edge device;
a system implemented using a robot;
a system incorporating one or more virtual machines (VMs);
a system implemented at least partially in a data center; or
a system implemented at least partially using cloud computing resources.