# US 12,007,728 B1 · Systems and methods for sensor data processing and object detection and motion prediction for robotic platforms

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,007,728 B1
- Document ID: US12007728B1
- Status: Granted
- Kind: B1
- Application number: 17/501,614
- Filed: 2021-10-14
- Granted: 2024-06-11
- Earliest claimed: 2020-10-14
- Assignees: Uatc, LLC
- Applicants: Uatc, LLC
- Inventors: Abhishek Mohta; Fang-Chieh Chou; Carlos Vallespi-Gonzalez; Brian C. Becker; Nemanja Djuric
- Web page: https://patentgrep.com/patent/US12007728B1

## CPC

- G05B 13/0265: {the criterion being a learning criterion}
- B60W 60/001: {Planning or execution of driving tasks}
- G06F 18/25: Fusion techniques
- G06F 30/27: using machine learning, e.g. artificial intelligence, neural networks, support vector machines [SVM] or training a model
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06N 3/09: Supervised learning
- G06V 10/80: Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level (multimodal speaker identification or verification G10L17/10)
- G06V 20/56: exterior to a vehicle by using sensors mounted on the vehicle
- G06V 20/58: Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
- B60W 50/00: Details of control systems for road vehicle drive control not related to the control of a particular sub-unit {, e.g. process diagnostic or vehicle driver interfaces}
- B60W 2050/0052: {Filtering, filters}
- B60W 2050/0083: {Setting, resetting, calibration}
- B60W 2420/403: Image sensing, e.g. optical camera
- B60W 2420/408: Radar; Laser, e.g. lidar
- B60W 2556/35: Data fusion
- G06F 2111/18: using virtual or augmented reality
- G05B: CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS (systems for controlling or regulating non-electric variables G05D; systems for regulating electric or magnetic variables G05F; control devices or systems insofar as characterised by mechanical features only G05G)
- B60W: CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

Systems and methods are disclosed for detecting and predicting the motion of objects within the surrounding environment of a system such as an autonomous vehicle. For example, an autonomous vehicle can obtain sensor data from a plurality of sensors comprising at least two different sensor modalities (e.g., RADAR, LIDAR, camera) and fused together to create a fused sensor sample. The fused sensor sample can then be provided as input to a machine learning model (e.g., a machine learning model for object detection and/or motion prediction). The machine learning model can have been trained by independently applying sensor dropout to the at least two different sensor modalities. Outputs received from the machine learning model in response to receipt of the fused sensor samples are characterized by improved generalization performance over multiple sensor modalities, thus yielding improved performance in detecting objects and predicting their future locations, as well as improved navigation performance.

## Claims

### Claim 1

A computer-implemented method for training an object detection model, comprising:
(a) obtaining sensor data from a plurality of sensors comprising at least two different sensor modalities;
(b) independently applying sensor dropout to the at least two different sensor modalities of the sensor data, including modifying intensity values of at least one of the at least two different sensor modalities;
(c) fusing the sensor data from the at least two different sensor modalities with sensor dropout independently applied thereto to generate a fused sensor sample;
(d) processing the fused sensor sample with at least a portion of the object detection model; and
(e) updating one or more weights of the object detection model based on labels associated with the sensor data; and
(f) employing the object detection model by a robotic platform operating within an environment.

### Claim 2 (depends on 1)

The computer-implemented method of claim 1, wherein (b) comprises independently applying sensor dropout to each of the at least two different sensor modalities at a fixed probability associated with the sensor modality.

### Claim 3 (depends on 1)

The computer-implemented method of claim 1, wherein the plurality of sensors comprise a RADAR system, a LIDAR system, and a camera.

### Claim 4 (depends on 3)

The computer-implemented method of claim 3, wherein:
the at least two different sensor modalities comprise at least one of the RADAR system or the camera; and
(b) comprises zeroing out a final feature vector for a portion of the sensor data obtained from the at least one of the RADAR system or the camera.

### Claim 5 (depends on 3)

The computer-implemented method of claim 3, wherein:
the at least two different sensor modalities comprise the LIDAR system; and
(b) comprises replacing a LIDAR intensity value with a sentinel value for a portion of the sensor data obtained from the LIDAR system.

### Claim 6 (depends on 2)

The computer-implemented method of claim 2, wherein the fixed probability associated with the sensor modality for at least one of the two different sensor modalities is zero.

### Claim 7 (depends on 1)

The computer-implemented method of claim 1, wherein the object detection model comprises an end-to-end model that is configured to jointly perform object detection and motion prediction.

### Claim 8 (depends on 1)

The computer-implemented method of claim 1, wherein the robotic platform comprises an autonomous vehicle.

### Claim 9 (depends on 1)

The computer-implemented method of claim 1, wherein the environment comprises a real-world environment or a simulated environment.

### Claim 10

An autonomous vehicle control system comprising:
one or more processors; and
one or more computer-readable medium storing instructions that when executed by the one or more processors cause the autonomous vehicle control system to perform operations, the operations comprising:
(a) obtaining sensor data from a plurality of sensors comprising at least two different sensor modalities;
(b) fusing the sensor data from the at least two different sensor modalities to create a fused sensor sample, wherein a sensor dropout is applied to the at least two different sensor modalities, the sensor dropout including modifying intensity values of at least one of the at least two different sensor modalities;
(c) providing the fused sensor sample as input to a machine learning model, the machine learning model having been trained by independently applying sensor dropout to the at least two different sensor modalities; and
(d) receiving, as an output of the machine learning model in response to receipt of the fused sensor sample provided as input, data indicative of one or more objects within an environment; and
(e) controlling an autonomous vehicle based on the data indicative of the one or more objects within the environment.

### Claim 11 (depends on 10)

The autonomous vehicle control system of claim 10, wherein the plurality of sensors comprise a RADAR system, a LIDAR system, and a camera.

### Claim 12 (depends on 11)

The autonomous vehicle control system of claim 11, wherein:
the sensor data is Obtained from at least one of the RADAR system or the camera; and
a training of the machine learning model comprises zeroing out a final feature vector for a portion of the sensor data obtained from the at least one of the RADAR system or the camera.

### Claim 13 (depends on 11)

The autonomous vehicle control system of claim 11, wherein:
the sensor data is obtained from the LIDAR system; and
a training of the machine learning model comprises replacing a LIDAR intensity value with a sentinel value for a portion of the sensor data obtained from the LIDAR system.

### Claim 14 (depends on 10)

The autonomous vehicle control system of claim 10, wherein the machine learning model comprises an end-to-end model that is configured to jointly perform object detection and motion prediction.

### Claim 15

An autonomous vehicle comprising:
one or more processors; and
one or more computer-readable medium storing instructions that when executed by the one or more processors cause the autonomous vehicle to perform operations, the operations comprising:
(a) obtaining sensor data from a plurality of sensors comprising at least two different sensor modalities;
(b) fusing the sensor data from the at least two different sensor modalities to create a fused sensor sample, wherein a sensor dropout is applied to the at least two different sensor modalities, the sensor dropout including modifying intensity values of at least one of the at least two different sensor modalities;
(c) providing the fused sensor sample as input to a machine learning model for object detection and motion prediction, the machine learning model for object detection and motion prediction having been trained by independently applying sensor dropout to the at least two different sensor modalities;
(d) receiving, as an output of the machine learning model for object detection and motion prediction in response to receipt of the fused sensor sample provided as input, perception data indicative of one or more states of one or more objects within an environment associated with the autonomous vehicle and prediction data indicative of one or more predicted future locations of the one or more objects;
(e) generating motion plan data based on the perception data and the prediction data; and
(f) controlling the autonomous vehicle based on the motion plan data.

### Claim 16 (depends on 15)

The autonomous vehicle of claim 15, wherein the plurality of sensors comprise a RADAR system, a LIDAR system, and a camera.

### Claim 17 (depends on 16)

The autonomous vehicle of claim 16, wherein:
the sensor data is obtained from at least one of the RADAR system or the camera; and
a training of the machine learning model for object detection and motion prediction comprises zeroing out a final feature vector for a portion of the sensor data Obtained from the at least one of the RADAR system or the camera.

### Claim 18 (depends on 16)

The autonomous vehicle of claim 16, wherein:
the sensor data is obtained from the LIDAR system; and
a training of the machine learning model for object detection and motion prediction comprises replacing a LIDAR intensity value with a sentinel value for a portion of the sensor data obtained from the LIDAR system.