US 11,688,181 B2 · Sensor fusion for autonomous machine applications using machine learning

Abstract

In various examples, a multi-sensor fusion machine learning model—such as a deep neural network (DNN)—may be deployed to fuse data from a plurality of individual machine learning models. As such, the multi-sensor fusion network may use outputs from a plurality of machine learning models as input to generate a fused output that represents data from fields of view or sensory fields of each of the sensors supplying the machine learning models, while accounting for learned associations between boundary or overlap regions of the various fields of view of the source sensors. In this way, the fused output may be less likely to include duplicate, inaccurate, or noisy data with respect to objects or features in the environment, as the fusion network may be trained to account for multiple instances of a same object appearing in different input representations.

Claims20

Dependent claims

A processor comprising:

one or more circuits to:
receive first data representative of a plurality of outputs of a plurality of deep neural networks (DNNs), at least one output of the plurality of outputs corresponding to a respective sensor having a respective field of view different from fields of view corresponding to one or more others sensors of a plurality of sensors of an autonomous machine;
compute, using a fusion DNN and based at least in part on the first data, second data representative of a fusion of the plurality of outputs;
and
perform one or more operations using the autonomous machine based at least in part on the second data.

2 more elements

A system comprising:

one or more processing units;
and
one or more memory units storing instructions that, when executed by the one or more processing units, cause the one or more processing units to execute operations comprising:
receiving first data representative of at least a first rasterized image generated using a first deep neural network (DNN) and based at least in part on first sensor data generated using a first sensor, the first rasterized image including at least a first object;
receiving second data representative of at least a second rasterized image generated using a second deep neural network (DNN) and based at least in part on second sensor data generated using a second sensor, the second rasterized image including at a least a second object;
computing, using a fusion DNN and based at least in part on the first data and the second data, third data representative of a fused rasterized image including both the first object and the second object;
and
performing one or more operations using an autonomous machine based at least in part on the third data.

5 more elements

Dependencies are inferred from the claim text.

Facts

Application

17/353,231Filed 21 Jun 2021 · us provisional application

Priority

25 Jun 2020

Published

US 2021/0406560 A130 Dec 2021

Granted

27 Jun 2023

Term ends

Not estimatedThe source records don’t support an estimate.

Classification47 codes

G06N 3/084
Learning methods: backpropagation, e.g. using gradient descent
G06V 20/588
Exterior to a vehicle by using sensors mounted on the vehicle: recognition of the road, e.g. of lane markings; Recognition of the vehicle driving…
B60W 60/001
Drive control systems specially adapted for autonomous road vehicles: planning or execution of driving tasks
B60W 60/00272
Using trajectory prediction for other traffic participants: relying on extrapolation of current movement

Sensor fusion for autonomous machine applications using machine learning

Life of this patent

Sensor fusion for autonomous machine applications using machine learning

Life of this patent

Abstract

Claims20