End-to-end tracking of objects

A computer-implemented method, comprising:

• receiving sensor data associated with an environment, the sensor data comprising RGB data and one or more LIDAR point clouds;
• determining an object detection associated with an object within the environment based at least in part on one or more first models and the sensor data;
• determining a detection score for the object detection, wherein determining a detection score comprises:
• fusing the one or more LIDAR point clouds with the RGB data by extracting one or more features from the RGB data;
• and
• encoding one or more binary parameters associated with the object detection, wherein at least one binary parameter of the one or more binary parameters is indicative of whether the object detection is associated with a trajectory beginning or a trajectory ending;
• the one or more first models comprising one or more first machine-learned models configured to detect the object in the environment based at least in part on the sensor data being input into the one or more first models and configured to determine the detection score for the object detection based on the one or more binary parameters;
• tracking the object detection over a sequence of sensor data inputs based at least in part on one or more second models,
• the one or more second models comprising one or more second machine-learned models configured to track the object over the sequence of sensor data inputs, the one or more second models being different from the one or more first models;
• and
• generating a trajectory for the object based at least in part on one or more linear constraints configured to link the object detection over the sequence of sensor data inputs.

An autonomous vehicle, comprising:

• a sensor system configured to generate sensor data of an environment external to the autonomous vehicle;
• and
• a vehicle computing system, comprising:
• one or more processors;
• and
• one or more non-transitory computer-readable media that store instructions for execution by the one or more processors to cause the vehicle computing system to perform operations, the operations comprising:
• receiving the sensor data, the sensor data comprising RGB data and one or more LIDAR point clouds;
• inputting the sensor data to one or more machine-learned models comprising one or more first models configured to detect one or more objects in the environment based at least in part on the sensor data and one or more second models configured to track the one or more objects over a sequence of sensor data inputs, the one or more second models being different from the one or more first models;
• determining, based at least in part on the one or more first models, an object detection associated with an object within the environment;
• determining a detection score for the object detection, wherein determining a detection score comprises:
• fusing the one or more LIDAR point clouds with the RGB data by extracting one or more features from the RGB data;
• and
• encoding one or more binary parameters associated with the object detection, wherein:
• the detection score for the object detection is based on the one or more binary parameters, and
• at least one binary parameter of the one or more binary parameters is indicative of whether the object detection is associated with a trajectory beginning or a trajectory ending;
• tracking, based at least in part on the one or more second models, the object detection over the sequence of sensor data inputs;
• and
• generating a trajectory for the object based at least in part on one or more linear constraints configured to link the object detection over the sequence of sensor data inputs.