End-to-end tracking of objects

A computer-implemented method of detecting objects of interest, comprising:

• receiving, by a computing system comprising one or more computing devices, sensor data from one or more sensors configured to generate sensor data associated with an environment;
• inputting, by the computing system, the sensor data to one or more machine-learned models including one or more first neural networks configured to detect one or more objects in the environment based at least in part on the sensor data and one or more second neural networks configured to track the one or more objects over a sequence of sensor data inputs;
• generating, by the computing system as an output of the one or more first neural networks, a three-dimensional (3D) bounding box and detection score for each of a plurality of object detections;
• generating, by the computing system as an output of the one or more second neural networks, a matching score associated with object detections over the sequence of sensor data inputs;
• generating, by the computing system using a flow network, a flow graph based at least in part on the matching scores;
• and
• determining, by the computing system using a linear program, a trajectory for each object detection based at least in part on the matching scores associated with the object detections over the sequence of sensor data inputs, wherein determining the trajectory for each object detection is based at least in part on the flow graph.