Systems and methods for end-to-end trajectory prediction using radar, LIDAR, and maps

A computer-implemented method for training a trajectory prediction network model from end-to-end, the network model comprising a plurality of subnetworks configured to extract one or more features from input data and a trajectory prediction machine-learned model, the method comprising:

• inputting training data into the network model, the training data comprising radar data comprising a plurality of sweeps, LIDAR data comprising a plurality of sweeps, and map data;
• generating, by the network model, one or more features for each of the LIDAR data;
• the radar data, and the map data, wherein generating the one or more features for the radar data comprises determining a feature vector for each cell in a coordinate frame and concatenating per sweep feature vectors for each cell;
• combining, by the network model, the one or more generated features for each of the LIDAR data, the radar data;
• and the map data to generate fused feature data;
• predicting, by the network model, a respective future trajectory for one or more objects detected by the network model;
• determining a loss function for the network model based at least in part on a comparison between the respective predicted future trajectory for the one or more Objects and a ground truth future trajectory;
• and training the network model based at least in part on the loss function.

An autonomous vehicle, comprising:

• one or more LIDAR sensors configured to obtain sweeps of LIDAR data;
• one or more radar sensors configured to obtain sweeps of radar data;
• and
• a computing system, comprising:
• one or more processors;
• a memory comprising one or more tangible non-transitory computer-readable media, the media storing computer-readable instructions that when executed by the one or more processors cause the one or more processors to perform operations comprising:
• obtaining, from the one or more LIDAR sensors, LIDAR data comprising one or more sweeps;
• obtaining, from the one or more radar sensors, radar data comprising one OF more sweeps;
• obtaining map data associated with the LIDAR data and the radar data;
• inputting the LIDAR data, the radar data, and the map data into a network model comprising one or more machine-learned models;
• generating, by the network model, one or more features for each of the LIDAR data, the radar data, and the map data, wherein generating the one or more features for the radar data comprises determining a feature vector for each cell in a coordinate frame and concatenating per sweep feature vectors for each cell;
• combining, by the network model, the one or more generated features for each of the LIDAR data, the radar data, and the map data to generate fused feature data;
• generating, by the network model, prediction data based at least in part on the fused feature data;
• receiving, as an output of the network model, the prediction data, the prediction data comprising a respective predicted trajectory for a future time period for one or more detected objects;
• determining a motion plan based at least in part on the respective predicted trajectory for the one or more detected objects;
• and
• controlling the autonomous vehicle based at least in part on the motion plan.