Systems and Methods for End-to-End Trajectory Prediction Using Radar, Lidar, and Maps

A computer-implemented method for training a trajectory prediction network model from end-to-end, the network model comprising a plurality of subnetworks configured to extract one or more features from input data and a trajectory prediction machine-learned model, the method comprising:

• inputting training data into the network model, the training data comprising radar data comprising a plurality of sweeps, LIDAR data comprising a plurality of sweeps, and map data;
• generating, by the network model, one or more features for each of the LIDAR data, the radar data, and the map data;
• combining, by the network model, the one or more generated features for each of the LIDAR data, the radar data, and the map data to generate fused feature data;
• predicting, by the network model, a respective future trajectory for one or more objects detected by the network model;
• determining a loss function for the network model based at least in part on a comparison between the respective predicted future trajectory for the one or more objects and a ground truth future trajectory;
• and
• training the network model based at least in part on the loss function.

An autonomous vehicle, comprising:

• one or more LIDAR sensors configured to obtain sweeps of LIDAR data;
• one or more radar sensors configured to obtain sweeps of radar data;
• and
• a computing system, comprising:
• one or more processors;
• a memory comprising one or more tangible non-transitory computer-readable media, the media storing computer-readable instructions that when executed by the one or more processors cause the one or more processors to perform operations comprising:
• obtaining, from the one or more LIDAR sensors, LIDAR data comprising one or more sweeps;
• obtaining, from the one or more radar sensors, radar data comprising one or more sweeps;
• obtaining map data associated with the LIDAR data and the radar data;
• inputting the LIDAR data, the radar data, and the map data into a network model comprising one or more machine-learned models;
• generating, by the network model, one or more features for each of the LIDAR data, the radar data, and the map data;
• combining, by the network model, the one or more generated features for each of the LIDAR data, the radar data, and the map data to generate fused feature data;
• generating, by the network model, prediction data based at least in part on the fused feature data;
• receiving, as an output of the network model, the prediction data, the prediction data comprising a respective predicted trajectory for a future time period for one or more detected objects;
• determining a motion plan based at least in part on the respective predicted trajectory for the one or more detected objects;
• and
• controlling the autonomous vehicle based at least in part on the motion plan.