# US 2022/0035376 A1 · Systems and Methods for End-to-End Trajectory Prediction Using Radar, Lidar, and Maps

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2022/0035376 A1
- Document ID: US20220035376A1
- Status: Published application
- Kind: A1
- Application number: 17/095,227
- Filed: 2020-11-11
- Published: 2022-02-03
- Earliest claimed: 2020-07-28
- Assignees: Uatc, LLC
- Applicants: Uatc, LLC
- Inventors: Ankit Laddah; Meet Pragnesh Shah; Zhiling Huang; Duncan Blake Barber; Matthew A. Langford; Carlos Vallespi-Gonzalez; Sida Zhang
- Web page: https://patentgrep.com/patent/US20220035376A1

## CPC

- G06N 3/08: Learning methods
- B60W 60/0027: {using trajectory prediction for other traffic participants}
- G01S 13/60: wherein the transmitter and receiver are mounted on the moving object, e.g. for determining ground speed, drift angle, ground track (G01S13/64 takes precedence)
- G01S 13/865: {Combination of radar systems with lidar systems}
- G01S 13/931: of land vehicles
- G05D 1/227: Handing over between remote control and on-board control; Handing over between remote control arrangements
- G05D 1/247: using signals provided by artificial sources external to the vehicle, e.g. navigation beacons
- G06F 18/24147: {Distances to closest patterns, e.g. nearest neighbour classification}
- G06F 18/251: {of input or preprocessed data}
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/09: Supervised learning
- G06N 5/041: {Abduction}
- G06N 20/00: Machine learning
- G06V 10/806: {of extracted features}
- G06V 10/82: using neural networks
- G06V 20/58: Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
- B60W 2420/408: Radar; Laser, e.g. lidar
- B60W 2556/40: High definition maps
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- B60W: CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
- G01S: RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G05D: SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

Systems and methods for trajectory prediction are provided. A method can include obtaining LIDAR data, radar data, and map data; inputting the LIDAR data, the radar data, and the map data into a network model; transforming, by the network model, the radar data into a coordinate frame associated with a most recent radar sweep in the radar data; generating, by the network model, one or more features for each of the LIDAR data, the transformed radar data, and the map data; combining, by the network model, the one or more generated features to generate fused feature data; generating, by the network model, prediction data based at least in part on the fused feature data; and receiving, as an output of the network model, the prediction data. The prediction data can include a respective predicted trajectory for a future time period for one or more detected objects.

## Claims

### Claim 1

A computing system, comprising:
one or more processors;
a memory comprising one or more tangible non-transitory computer-readable media, the media storing computer-readable instructions that when executed by the one or more processors cause the one or more processors to perform operations comprising:
obtaining, from one or more sensors of an autonomous vehicle, LIDAR data and radar data, the radar data comprising one or more radar sweeps, the LIDAR data comprising one or more LIDAR sweeps;
inputting the LIDAR data and the radar data into a network model comprising one or more machine-learned models;
generating, by the network model, one or more features for each of the LIDAR data and the radar data;
combining, by the network model, the one or more generated features for each of the LIDAR data and the radar data to generate fused feature data;
generating, by the network model, prediction data based at least in part on the fused feature data; and
receiving, as an output of the network model, the prediction data, the prediction data comprising a respective predicted trajectory for a future time period for one or more detected objects;
wherein the one or more detected objects may optionally be received as an output of the network model.

### Claim 2 (depends on 1)

The computing system of claim 1, wherein the operations further comprise:
determining a motion plan based at least in part on the respective predicted trajectory for the one or more detected objects; and
controlling an autonomous vehicle based at least in part on the motion plan.

### Claim 3 (depends on 1)

The computing system of claim 1, wherein combining, by the network model, the one or more generated features for each of the LIDAR data and the radar data to generate the fused feature data comprises:
receiving, as an output of a respective subnetwork of the network model, the one or more respective generated features for each of the LIDAR data and the radar data;
inputting the one or more respective generated features for the LIDAR data and the radar data into a multi-scale fusion backbone of the network model; and
concatenating, by the multi-scale fusion backbone, the one or more respective generated features for the LIDAR data and the radar data to generate the fused feature data.

### Claim 4 (depends on 1)

The computing system of claim 1, wherein the operations further comprise:
transforming the radar data into a coordinate frame associated with a most recent radar sweep in the radar data, the coordinate frame comprising a plurality of cells.

### Claim 5 (depends on 5)

The computing system of claim 5, wherein the coordinate frame comprises a two-dimensional top-down grid comprising a cartesian grid or an arc grid.

### Claim 6 (depends on 1)

The computing system of claim 1, wherein generating, by the network model, one or more features for the radar data comprises:
determining a feature vector for each cell in the coordinate frame for each sweep in the radar data; and
concatenating the per sweep feature vectors for each cell.

### Claim 7 (depends on 6)

The computing system of claim 6, wherein the feature vector for each cell in the coordinate frame for each sweep in the radar data is further determined using a graph between the cell and the radar data using a nearest neighbor algorithm.

### Claim 8 (depends on 1)

The computing system of claim 1, wherein generating, by the network model, the one or more features for the LIDAR data comprises:
discretizing the LIDAR data points into an occupancy coordinate frame comprising a plurality of cells for each sweep; and
concatenating the per sweep occupancy features for each cell.

### Claim 9 (depends on 1)

The computing system of claim 1, wherein the operations further comprise:
obtaining map data associated with the LIDAR data and the radar data; and
generating, by the network model, one or more features for the map data comprises rasterizing the map data into a coordinate frame comprising a plurality of cells; and
wherein the fused feature data is generated by combining the one or more features for the map data with the one or more features for each of the LIDAR data and the radar data.

### Claim 10 (depends on 1)

The computing system of claim 1, wherein the network model has been trained form end-to-end using a loss function comprising a detection loss parameter and a trajectory loss parameter.

### Claim 11 (depends on 10)

The computing system of claim 10, wherein the detection loss parameter comprises a classification loss parameter and a regression loss parameter.

### Claim 12 (depends on 1)

The computing system of claim 1, wherein the prediction data comprises a probabilistic multi-hypothesis trajectory prediction for each of the one or more detected objects.

### Claim 13 (depends on 1)

The computing system of claim 1, wherein the prediction data comprises a single hypothesis trajectory prediction for each of the one or more detected objects.

### Claim 14 (depends on 1)

The computing system of claim 1, wherein generating, by the network model, the prediction data based at least in part on the fused feature data comprises inputting the fused feature data into a prediction head of the network model.

### Claim 15

A computer-implemented method for training a trajectory prediction network model from end-to-end, the network model comprising a plurality of subnetworks configured to extract one or more features from input data and a trajectory prediction machine-learned model, the method comprising:
inputting training data into the network model, the training data comprising radar data comprising a plurality of sweeps, LIDAR data comprising a plurality of sweeps, and map data;
generating, by the network model, one or more features for each of the LIDAR data, the radar data, and the map data;
combining, by the network model, the one or more generated features for each of the LIDAR data, the radar data, and the map data to generate fused feature data;
predicting, by the network model, a respective future trajectory for one or more objects detected by the network model;
determining a loss function for the network model based at least in part on a comparison between the respective predicted future trajectory for the one or more objects and a ground truth future trajectory; and
training the network model based at least in part on the loss function.

### Claim 16 (depends on 15)

The computer-implemented method for training the trajectory prediction network model from end-to-end of claim 15, wherein the loss function comprises a detection loss parameter and a trajectory loss parameter.

### Claim 17 (depends on 16)

The computer-implemented method for training the trajectory prediction network model from end-to-end of claim 16, wherein the detection loss parameter comprises a classification loss parameter and a regression loss parameter.

### Claim 18 (depends on 15)

The computer-implemented method for training the trajectory prediction network model from end-to-end of claim 15, wherein the training data comprises LIDAR data and radar data acquired during one or more autonomous vehicle driving sessions.

### Claim 19

An autonomous vehicle, comprising:
one or more LIDAR sensors configured to obtain sweeps of LIDAR data;
one or more radar sensors configured to obtain sweeps of radar data; and
a computing system, comprising:
one or more processors;
a memory comprising one or more tangible non-transitory computer-readable media, the media storing computer-readable instructions that when executed by the one or more processors cause the one or more processors to perform operations comprising:
obtaining, from the one or more LIDAR sensors, LIDAR data comprising one or more sweeps;
obtaining, from the one or more radar sensors, radar data comprising one or more sweeps;
obtaining map data associated with the LIDAR data and the radar data;
inputting the LIDAR data, the radar data, and the map data into a network model comprising one or more machine-learned models;
generating, by the network model, one or more features for each of the LIDAR data, the radar data, and the map data;
combining, by the network model, the one or more generated features for each of the LIDAR data, the radar data, and the map data to generate fused feature data;
generating, by the network model, prediction data based at least in part on the fused feature data;
receiving, as an output of the network model, the prediction data, the prediction data comprising a respective predicted trajectory for a future time period for one or more detected objects;
determining a motion plan based at least in part on the respective predicted trajectory for the one or more detected objects; and
controlling the autonomous vehicle based at least in part on the motion plan.

### Claim 20 (depends on 19)

The autonomous vehicle of claim 19, wherein generating, by the network model, the one or more features for the radar data comprises:
transforming the radar data into a coordinate frame associated with a most recent radar sweep in the radar data, the coordinate frame comprising a plurality of cells;
determining a feature vector for each cell in the coordinate frame for each sweep in the radar data; and
concatenating the per sweep feature vectors for each cell.
