US 11,906,660 B2 · Object detection and classification using LiDAR range images for autonomous machine applications

Abstract

In various examples, a deep neural network (DNN) may be used to detect and classify animate objects and/or parts of an environment. The DNN may be trained using camera-to-LiDAR cross injection to generate reliable ground truth data for LiDAR range images. For example, annotations generated in the image domain may be propagated to the LiDAR domain to increase the accuracy of the ground truth data in the LiDAR domain—e.g., without requiring manual annotation in the LiDAR domain. Once trained, the DNN may output instance segmentation masks, class segmentation masks, and/or bounding shape proposals corresponding to two-dimensional (2D) LiDAR range images, and the outputs may be fused together to project the outputs into three-dimensional (3D) LiDAR point clouds. This 2D and/or 3D information output by the DNN may be provided to an autonomous vehicle drive stack to enable safe planning and control of the autonomous vehicle.

Claims19

Dependent claims

A method comprising:

applying, to a deep neural network (DNN), first data corresponding to a range image, the range image being generated using sensor data from a LiDAR point cloud;
computing, using the DNN and for the range image corresponding to the first data, one or more segmentation masks representing one or more portions of the range image that correspond to one or more objects;
computing, using the DNN and for the range image corresponding to the first data, one or more locations within the range image of one or more bounding shape proposals corresponding to the one or more objects;
correlating the one or more bounding shape proposals for the range image with the one or more segmentation masks for the range image;
based at least on the correlating, generating one or more projected bounding shape proposals by projecting the one or more locations of the one or more bounding shape proposals within the range image to one or more corresponding locations in the LiDAR point cloud;
and
performing one or more operations using a machine based at least on the one or more projected bounding shape proposals.

4 more elements

A system comprising:

one or more processing units to:
compute, using one or more neural networks and based at least on a range image that is generated using a LiDAR point cloud, first data representing one or more segmentation masks corresponding to one or more portions of the range image that are associated with one or more objects;
compute, using the one or more neural networks and based at least on the range image, second data representing one or more locations within the range image of one or more bounding shape proposals corresponding to the one or more objects;
correlate the one or more bounding shape proposals with the one or more segmentation masks;
generate, based at least on the first data and the second data, one or more projected bounding shape proposals based at least on projecting the one or more locations of the one or more bounding shape proposals within the range image to one or more corresponding locations in the LiDAR point cloud;
and
perform one or more operations using a machine based at least on the one or more projected bounding shape proposals.

4 more elements

A processor comprising:

one or more processing units to:
compute, using one or more neural networks and for a LiDAR range image, one or more segmentation masks corresponding to one or more portions of the LiDAR range image that are associated with one or more objects;
compute, using the one or more neural networks and for the LiDAR range image, one or more locations within the LIDAR range image of one or more bounding shape proposals corresponding to the one or more objects;
correlate the one or more bounding shape proposals with the one or more segmentation masks;
generate, based at least on the one or more segmentation masks and the one or more bounding shape proposals, one or more projected bounding shape proposals based at least on projecting the one or more locations of the one or more bounding shape proposals within the LIDAR range image to one or more corresponding locations in a LiDAR point cloud;
and
perform one or more operations using a machine based at least on the one or more projected bounding shape proposals.

4 more elements

Dependencies are inferred from the claim text.

Object detection and classification using LiDAR range images for autonomous machine applications

Life of this patent

Object detection and classification using LiDAR range images for autonomous machine applications

Life of this patent

Abstract

Claims19