US 12,007,728 B1 · Systems and methods for sensor data processing and object detection and motion prediction for robotic platforms

Abstract

Systems and methods are disclosed for detecting and predicting the motion of objects within the surrounding environment of a system such as an autonomous vehicle. For example, an autonomous vehicle can obtain sensor data from a plurality of sensors comprising at least two different sensor modalities (e.g., RADAR, LIDAR, camera) and fused together to create a fused sensor sample. The fused sensor sample can then be provided as input to a machine learning model (e.g., a machine learning model for object detection and/or motion prediction). The machine learning model can have been trained by independently applying sensor dropout to the at least two different sensor modalities. Outputs received from the machine learning model in response to receipt of the fused sensor samples are characterized by improved generalization performance over multiple sensor modalities, thus yielding improved performance in detecting objects and predicting their future locations, as well as improved navigation performance.

Claims18

Dependent claims

A computer-implemented method for training an object detection model, comprising:

(a) obtaining sensor data from a plurality of sensors comprising at least two different sensor modalities;
(b) independently applying sensor dropout to the at least two different sensor modalities of the sensor data, including modifying intensity values of at least one of the at least two different sensor modalities;
(c) fusing the sensor data from the at least two different sensor modalities with sensor dropout independently applied thereto to generate a fused sensor sample;
(d) processing the fused sensor sample with at least a portion of the object detection model;
and
(e) updating one or more weights of the object detection model based on labels associated with the sensor data;
and
(f) employing the object detection model by a robotic platform operating within an environment.

5 more elements

An autonomous vehicle control system comprising:

one or more processors;
and
one or more computer-readable medium storing instructions that when executed by the one or more processors cause the autonomous vehicle control system to perform operations, the operations comprising:
(a) obtaining sensor data from a plurality of sensors comprising at least two different sensor modalities;
(b) fusing the sensor data from the at least two different sensor modalities to create a fused sensor sample, wherein a sensor dropout is applied to the at least two different sensor modalities, the sensor dropout including modifying intensity values of at least one of the at least two different sensor modalities;
(c) providing the fused sensor sample as input to a machine learning model, the machine learning model having been trained by independently applying sensor dropout to the at least two different sensor modalities;
and
(d) receiving, as an output of the machine learning model in response to receipt of the fused sensor sample provided as input, data indicative of one or more objects within an environment;
and
(e) controlling an autonomous vehicle based on the data indicative of the one or more objects within the environment.

7 more elements

An autonomous vehicle comprising:

one or more processors;
and
one or more computer-readable medium storing instructions that when executed by the one or more processors cause the autonomous vehicle to perform operations, the operations comprising:
(a) obtaining sensor data from a plurality of sensors comprising at least two different sensor modalities;
(b) fusing the sensor data from the at least two different sensor modalities to create a fused sensor sample, wherein a sensor dropout is applied to the at least two different sensor modalities, the sensor dropout including modifying intensity values of at least one of the at least two different sensor modalities;
(c) providing the fused sensor sample as input to a machine learning model for object detection and motion prediction, the machine learning model for object detection and motion prediction having been trained by independently applying sensor dropout to the at least two different sensor modalities;
(d) receiving, as an output of the machine learning model for object detection and motion prediction in response to receipt of the fused sensor sample provided as input, perception data indicative of one or more states of one or more objects within an environment associated with the autonomous vehicle and prediction data indicative of one or more predicted future locations of the one or more objects;
(e) generating motion plan data based on the perception data and the prediction data;
and
(f) controlling the autonomous vehicle based on the motion plan data.

7 more elements

Dependencies are inferred from the claim text.

Systems and methods for sensor data processing and object detection and motion prediction for robotic platforms

Life of this patent

Systems and methods for sensor data processing and object detection and motion prediction for robotic platforms

Life of this patent

Abstract

Claims18