US 12,008,822 B2 · Using neural networks for 3D surface structure estimation based on real-world data for autonomous systems and applications

Abstract

In various examples, to support training a deep neural network (DNN) to predict a dense representation of a 3D surface structure of interest, a training dataset is generated from real-world data. For example, one or more vehicles may collect image data and LiDAR data while navigating through a real-world environment. To generate input training data, 3D surface structure estimation may be performed on captured image data to generate a sparse representation of a 3D surface structure of interest (e.g., a 3D road surface). To generate corresponding ground truth training data, captured LiDAR data may be smoothed, subject to outlier removal, subject to triangulation to filling missing values, accumulated from multiple LiDAR sensors, aligned with corresponding frames of image data, and/or annotated to identify 3D points on the 3D surface of interest, and the identified 3D points may be projected to generate a dense representation of the 3D surface structure.

Claims21

Dependent claims

A method comprising:

generating, based at least on image data captured during a capture session in an environment, a first representation of a three-dimensional (3D) surface structure of a component of the environment;
generating, based at least on LiDAR data associated with the image data and captured during the capture session, a second representation of the 3D surface structure of the component;
and
training one or more neural networks (NNs) to generate a densified representation of the 3D surface Lure using the first representation of the 3D surface structure as input training data and using the second representation of the 3D surface structure as around truth training data.

1 more element

A system comprising:

one or more processing units;
and
one or more memory units storing instructions that, when executed by the one or more processing units, cause the one or more processing units to execute operations comprising:
capturing image data and LiDAR data during a capture session in an environment;
generating, based at least on applying 3D structure estimation to the image data, an incomplete representation of a three-dimensional (3D) surface structure of a surface in the environment;
generating, based at least on labeling of the LiDAR data, a ground truth representation of the 3D surface structure of the surface;
and
including the incomplete representation and the ground truth representation of the 3D surface structure in a training dataset.

5 more elements

Dependencies are inferred from the claim text.

Using neural networks for 3D surface structure estimation based on real-world data for autonomous systems and applications

Life of this patent

Using neural networks for 3D surface structure estimation based on real-world data for autonomous systems and applications

Life of this patent

Abstract

Claims21