# US 2021/0063578 A1 · Object detection and classification using lidar range images for autonomous machine applications

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0063578 A1
- Document ID: US20210063578A1
- Status: Published application
- Kind: A1
- Application number: 17/005,788
- Filed: 2020-08-28
- Published: 2021-03-04
- Earliest claimed: 2019-08-30
- Assignees: NVIDIA Corporation
- Applicants: NVIDIA Corporation
- Inventors: Tilman Wekel; Sangmin Oh; David Nister; Joachim Pehserl; Neda Cvijetic; Ibrahim Eden
- Web page: https://patentgrep.com/patent/US20210063578A1

## CPC

- G01S 7/4802: {using analysis of echo signal for target characterisation; Target signature; Target cross-section}
- G01S 7/481: Constructional features, e.g. arrangements of optical elements
- G01S 13/862: {Combination of radar systems with sonar systems}
- G01S 13/865: {Combination of radar systems with lidar systems}
- G01S 13/867: {Combination of radar systems with cameras}
- G01S 13/931: of land vehicles
- G01S 17/86: Combinations of lidar systems with systems other than lidar, radar or sonar, e.g. with direction finders
- G01S 17/894: Three-dimensional [3D] imaging with simultaneous measurement of time-of-flight at a two-dimensional [2D] array of receiver pixels, e.g. time-of-flight cameras or flash lidar
- G01S 17/931: of land vehicles
- G06F 18/25: Fusion techniques
- G06V 10/764: using classification, e.g. of video objects
- G06V 10/80: Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level (multimodal speaker identification or verification G10L17/10)
- G06V 10/82: using neural networks
- G06V 20/58: Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
- G01S 7/28: Details of pulse systems
- G01S 2013/9316: {combined with communication equipment with other vehicles or with base stations}
- G01S: RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

In various examples, a deep neural network (DNN) may be used to detect and classify animate objects and/or parts of an environment. The DNN may be trained using camera-to-LiDAR cross injection to generate reliable ground truth data for LiDAR range images. For example, annotations generated in the image domain may be propagated to the LiDAR domain to increase the accuracy of the ground truth data in the LiDAR domain—e.g., without requiring manual annotation in the LiDAR domain. Once trained, the DNN may output instance segmentation masks, class segmentation masks, and/or bounding shape proposals corresponding to two-dimensional (2D) LiDAR range images, and the outputs may be fused together to project the outputs into three-dimensional (3D) LiDAR point clouds. This 2D and/or 3D information output by the DNN may be provided to an autonomous vehicle drive stack to enable safe planning and control of the autonomous vehicle.

## Claims

### Claim 1

A method comprising:
applying, to a deep neural network (DNN), first data corresponding to a range image, the range image being generated using sensor data from a LiDAR point cloud;
computing, using the DNN and based at least in part on the first data, one or more segmentation masks corresponding to one or more objects detected in the range image;
computing one or more bounding shape proposals corresponding to the one or more objects;
correlating the one or more bounding shape proposals with the one or more segmentation masks;
based at least in part on the correlating, generating projected bounding shape proposals by projecting the one or more bounding shape proposals to the LiDAR point cloud; and
performing one or more operations by an autonomous machine based at least in part on the projected bounding shape proposals.

### Claim 2 (depends on 1)

The method of claim 1, wherein:
the first data is applied to the DNN as at least one of a first channel corresponding to a distance, a second channel corresponding to an elevation, or a third channel corresponding to an intensity; and
at least one of the distance, the elevation, or the intensity are determined from the range image.

### Claim 3 (depends on 1)

The method of claim 1, wherein the one or more segmentation masks comprises at least at least one of:
an instance segmentation mask representative of unique instances of the one or more objects; or
a semantic segmentation mask representative of classifications of the one or more objects.

### Claim 4 (depends on 3)

The method of claim 3, wherein at least one of:
the instance segmentation mask includes first values for each pixel of the range image determined to correspond to the one or more objects; or
the semantic segmentation mask includes second values for each pixel of the range image determined to correspond to the one or more objects.

### Claim 5 (depends on 1)

The method of claim 1, further comprising:
generating, using a LiDAR sensor of a vehicle, LiDAR data representative of the LiDAR point cloud corresponding to a sensory field of the LiDAR sensor; and
generating the range image based at least in part on the LiDAR point cloud.

### Claim 6 (depends on 1)

The method of claim 1, wherein the projecting the bounding shape proposals to the LiDAR point cloud includes:
determining, based at least in part on the range image, depth values corresponding to each pixel associated with each bounding shape proposal of the bounding shape proposals;
determining, based at least in part on the depth values, a centroid of an object of the objects corresponding to the bounding shape; and
projecting the bounding shape proposal at a location of the centroid within the LiDAR point cloud.

### Claim 7 (depends on 1)

The method of claim 1, wherein the DNN is trained using camera-to-LiDAR cross-injection.

### Claim 8 (depends on 1)

The method of claim 1, wherein the camera-to-LiDAR cross-injection includes:
transferring ground truth data from a training image to a training LiDAR point cloud captured substantially simultaneously with the training image;
applying a geometric constraint to the transferred ground truth data to generate updated ground truth data corresponding to the training LiDAR point cloud; and
projecting the updated ground truth data to a training LiDAR range image to generate final ground truth data.

### Claim 9

A method comprising:
receiving image data representative of a scene captured at a time using an image sensor and LiDAR data representative of a LiDAR point cloud captured at the time using a LiDAR sensor, the image sensor and the LiDAR sensor having at least partially overlapping sensory fields;
generating ground truth data corresponding to annotations associated with the image data;
transforming the ground truth data to the LiDAR point cloud to generate transformed ground truth data;
applying a geometric constraint to the transformed ground truth data to generate updated ground truth data;
generating final ground truth data based at least in part on projecting the updated ground truth data to a range image generated based at least in part on the LiDAR point cloud; and
using the final ground truth data to train a deep neural network (DNN) to predict one or more outputs using data representative of the range image as input.

### Claim 10 (depends on 9)

The method of claim 9, wherein the annotations include at least one of bounding shapes corresponding to objects represented in the image data, first contour lines corresponding to regions represented in the image data associated with semantic classes, or second contour lines corresponding to unique object instances of the objects.

### Claim 11 (depends on 10)

The method of claim 10, wherein the final ground truth data is representative of at least one of first values corresponding to at least one of a location or geometry of the bounding shapes, second values representative of semantic class associations for each pixel within the first contour lines, or third values representative of instance identifiers for each pixel within the second contour lines.

### Claim 12 (depends on 9)

The method of claim 9, wherein the applying the geometric constraint includes executing a random sample consensus (RANSAC) algorithm.

### Claim 13 (depends on 9)

The method of claim 9, wherein the data representative of the LiDAR range image corresponds to at least one of a distance, an elevation, or an intensity of each pixel of the LiDAR range image.

### Claim 14 (depends on 9)

The method of claim 9, further comprising:
positioning a virtual LiDAR sensor within the LiDAR point cloud;
generating an additional range image from a perspective of the virtual LiDAR sensor using simulated LiDAR sensor data;
generating additional final ground truth data based at least in part on projecting at least a subset of the updated ground truth data within a virtual sensory field of the virtual LiDAR sensor to the additional range image,
wherein additional final ground truth data is further used to train the DNN.

### Claim 15 (depends on 9)

The method of claim 9, wherein a first sensory field of the image sensor corresponds to a first portion of a second sensory field of the LiDAR sensor, and additional image data representative of an additional image generated by an additional image sensor having a third sensory field corresponding to a second portion of the second sensory field of the LiDAR sensor is used to generate additional ground truth data for training the DNN.

### Claim 16 (depends on 9)

The method of claim 9, wherein the method is performed, at least in part, using one more virtual machines.

### Claim 17

A system comprising:
an image sensor;
a LiDAR sensor;
one or more processors;
one or more memory devices storing instructions thereon that, when executed using the one or more processors, cause the one or more processors execute operations comprising:
receiving image data representative of an image captured at a time using the image sensor and LiDAR data representative of a three-dimensional (3D) LiDAR representation captured at the time using the LiDAR sensor;
generating initial ground truth data corresponding to annotations associated with the image;
unprojecting the initial ground truth data to the 3D LiDAR representation to generate transformed ground truth data;
generating final ground truth data based at least in part on projecting the updated ground truth data to a two-dimensional (2D) LiDAR representation generated based at least in part on the 3D LiDAR representation; and
training a deep neural network (DNN) using the final ground truth data.

### Claim 18 (depends on 17)

The system of claim 17, wherein the annotations include at least one of bounding shapes corresponding to objects depicted in the image, first contour lines corresponding to regions of the image associated with semantic classes, or second contour lines corresponding to unique object instances of the objects.

### Claim 19 (depends on 18)

The system of claim 18, wherein the initial ground truth data is representative of at least one of first values corresponding to at least one of a location or geometry of the bounding shapes, second values representative of semantic class associations for each pixel within the first contour lines, or third values representative of instance identifiers for each pixel within the second contour lines.

### Claim 20 (depends on 17)

The system of claim 17, wherein the 2D LiDAR representation corresponds to a LiDAR range image and the 3D LiDAR representation corresponds to a LiDAR point cloud.

### Claim 21 (depends on 17)

The system of claim 17, wherein the operations further comprise:
positioning a virtual LiDAR sensor within the 3D LiDAR representation;
generating an additional 2D LiDAR representation from a perspective of the virtual LiDAR sensor;
generating additional final ground truth data based at least in part on projecting at least a subset of the updated ground truth data within a virtual sensory field of the virtual LiDAR sensor to the additional 2D LiDAR representation,
wherein the additional final ground truth data is further used to train the DNN.
