US 12,755,521 B2Granted

Object detection using sensor fusion for autonomous systems and applications

AssigneeNVIDIA Corporation

InventorsNeeraj Sajjan, Mehmet K. Kocamaz

Life of this patent

Estimating the term…

Loading patent…

Abstract

In various examples, sensor fusion for autonomous or semi-autonomous systems and applications is described. Systems and methods are disclosed that use data pipelines to process sensor data generated using different types of sensors (e.g., image sensors, RADAR sensors, LiDAR sensors, etc.) in order to generate first data representing information associated with objects surrounding a vehicle. For instance, the information may represent values for parameters associated with the objects, such as locations of the objects, dimensions of the objects, velocities of the objects, orientations of the objects, classifications of the objects, and/or any of parameters. The systems and methods may then process the first data using one or more machine learning models (e.g., one or more deep neural networks) that are trained to fuse the information (e.g., the parameters) and output second data representing final information associated with the objects. The fused output may then be used to perform downstream operations.

Claims20

Dependent claims

A method comprising:

determining, based at least on first sensor data obtained using a first type of sensor, first values for parameters associated with an object located within an environment at a first instance in time;
determining, based at least on second sensor data obtained using a second type of sensor, second values for the parameters associated with the object at a second instance in time;
generating input data representative of at least:
a plurality of first grids associated with the first type of sensor, the plurality of first grids indicating the first values for the parameters within first portions of the plurality of first grids that are associated with a location of the object within the environment;
and
a plurality of second grids associated with the second type of sensor, the plurality of second grids indicating the second values for the parameters within second portions of the plurality of second grids that are associated with the location of the object within the environment;
determining, using one or more neural networks and based at least on the input data, third values for the parameters associated with the object;
and
causing a machine to navigate within the environment based at least on the third values for the parameters.

6 more elements

A system comprising:

one or more processors to:
receive first data representative of first values for parameters associated with an object, the first data obtained using first sensor data associated with a first type of sensor;
receive second data representative of second values for the parameters associated with the object, the second data obtained using second sensor data associated with a second type of sensor;
generate input data representative of at least:
a plurality of first grids, an individual first grid of the plurality of first grids being associated with an individual parameter of the parameters and indicating an individual first value of the first values for the individual parameter;
and
a plurality of second grids, an individual grid of the plurality of second grids being associated with the individual parameter of the parameters and indicating an individual second value of the second values for the individual parameter;
determine, using one or more neural networks and based at least on the input data, third values for the parameters associated with the object;
and
cause a machine to navigate within an environment based at least on the third values.

7 more elements

A processor comprising processing circuitry to:

receive first data representative of first values for parameters associated with an object, the first data obtained using first sensor data associated with a first type of sensor;
receive second data representative of second values for the parameters associated with the object, the second data obtained using second sensor data associated with a second type of sensor;
generate input data representative of at least:
a plurality of first grids, an individual first grid of the plurality of first grids being associated with an individual parameter of the parameters and indicating an individual first value of the first values for the individual parameter;
and
a plurality of second grids, an individual grid of the plurality of second grids being associated with the individual parameter of the parameters and indicating an individual second value of the second values for the individual parameter;
determine, using one or more neural networks and based at least on the input data, third values for the parameters associated with the object;
and
cause a machine to navigate within an environment based at least on the e third values.

6 more elements

Dependencies are inferred from the claim text.