US 11,960,026 B2Granted

Deep neural network for detecting obstacle instances using radar sensors in autonomous machine applications

AssigneeNVIDIA Corporation

InventorsAlexander Popov, Nikolai Smolyanskiy, Ryan Oldja, Shane Murray, Tilman Wekel, David Nister, Joachim Pehserl, Ruchi Bhargava, Sangmin Oh

Life of this patent

Estimating the term…

patentgrep

Loading patent…

Abstract

In various examples, a deep neural network(s) (e.g., a convolutional neural network) may be trained to detect moving and stationary obstacles from RADAR data of a three dimensional (3D) space. In some embodiments, ground truth training data for the neural network(s) may be generated from LIDAR data. More specifically, a scene may be observed with RADAR and LIDAR sensors to collect RADAR data and LIDAR data for a particular time slice. The RADAR data may be used for input training data, and the LIDAR data associated with the same or closest time slice as the RADAR data may be annotated with ground truth labels identifying objects to be detected. The LIDAR labels may be propagated to the RADAR data, and LIDAR labels containing less than some threshold number of RADAR detections may be omitted. The (remaining) LIDAR labels may be used to generate ground truth data.

Claims20

Dependent claims

A processor comprising:

one or more processing units to perform one or more operations based at least on one or more outputs of a neural network, the neural network being trained, at least, by:
receiving (i) RADAR data collected from a three-dimensional (3D) environment using a RADAR sensor and (ii) 3D data collected from the 3D environment using a different type of sensor than the RADAR sensor;
projecting a first instance of the 3D data associated with a time slice to generate a first projection image representative of one or more 3D detections from the different type of sensor within the 3D environment;
receiving one or more labels representative of the one or more 3D detections from the different type of sensor, the one or more labels identifying one or more locations in the first projection image corresponding to one or more objects represented by the one or more 3D detections;
projecting a second instance of the RADAR data associated with the time slice to generate a RADAR projection image representative of one or more RADAR detections within the 3D environment;
and
updating one or more parameters of the neural network based at least on the RADAR projection image and ground truth data generated based at least on the one or more labels.

4 more elements

A system comprising:

one or more processing units to perform one or more operations based at least on one or more outputs of a neural network, the neural network being trained, at least, by:
receiving (i) RADAR data collected using a RADAR sensor and (ii) three-dimensional (3D) data collected using a different type of sensor than the RADAR sensor;
for at least one frame of one or more frames of the 3D data:
generating, using the at least one frame of 3D data, a first projection image representative of one or more 3D detections from the different type of sensor;
receiving one or more labels representative of the one or more 3D detections from the different type of sensor, the one or more labels identifying one or more locations in the first projection image corresponding to one or more objects represented by the one or more 3D detections;
generating, using a corresponding frame of the RADAR data associated with the at least one frame of 3D data, a RADAR projection image representative of one or more RADAR detections;
and
updating one or more parameters of the neural network based at least on the RADAR projection image and ground truth data generated based at least on the one or more labels.

5 more elements

Dependencies are inferred from the claim text.

Facts

Application: 17/976,581Filed 28 Oct 2022 · continuation
Priority: 21 Nov 2019
Published: US 2023/0049567 A116 Feb 2023
Granted: 16 Apr 2024
Term ends: Not estimatedThe source records don’t support an estimate.

Classification7 codes

G01S 7/417
Using analysis of echo signal for target characterisation; Target signature; Target cross-section: involving the use of neural networks
G01S 13/865
Combinations of radar systems with non-radar systems, e.g. sonar, direction finder: combination of radar systems with lidar systems
G01S 13/89
Radar or analogous systems specially adapted for specific applications: for mapping or imaging
G06N 3/04
Neural networks: architecture, e.g. interconnection topology

Family

Application62/938,852Us provisional application · filed 21 Nov 2019
Application16/836,618Parent of this continuation · filed 31 Mar 2020
US 2023/0049567 A1Pre-grant publication · published 16 Feb 2023
US 11,531,088 B2Parent of this continuation
US 12,050,285 B2Related patent
US 2021/0156963 A1Published application
US 11,960,026 B2This patent

From the US filing history. Foreign family members aren’t listed.