Perception and understanding of road users and road objects

A computer-implemented method for understanding road users and road objects and controlling a vehicle based on the understanding, the method comprising:

• determining, by a tracker, tracked objects in an environment of the vehicle;
• determining, by a main understanding model, that a tracked object has an unknown object classification;
• providing sensor data corresponding to the tracked object having the unknown object classification to a sub-model;
• determining, by the sub-model, a plurality of inferences based on the sensor data, wherein determining the plurality of inferences comprises:
• processing the sensor data using a shared backbone;
• and
• generating inferences by a plurality of heads that are downstream of the backbone, the inferences including one or more road object classifications and one or more road object attributes;
• providing the inferences to a tracker that collects the inferences of the tracked objects and a prediction part that predicts behaviors of the tracked objects;
• and
• planning a trajectory of the vehicle based on tracked objects information from the tracker and predictions from the prediction part.

One or more non-transitory computer-readable media storing instructions that, when executed by one or more processors, cause the one or more processors to:

• determine, by a tracker encoded in the instructions, tracked objects in an environment of a vehicle;
• determine, by a main understanding model encoded in the instructions, that a tracked object has an unknown object classification;
• provide sensor data corresponding to the tracked object having the unknown object classification to a sub-model encoded in the instructions;
• determine, by the sub-model, a plurality of inferences based on the sensor data, wherein determining the plurality of inferences comprises:
• processing the sensor data using a shared backbone encoded in the instructions;
• and
• generating inferences by a plurality of heads encoded in the instructions that are downstream of the backbone, the inferences including one or more road object classifications and one or more road object attributes;
• provide the inferences to a tracker encoded in the instructions that collects the inferences of the tracked objects and a prediction part encoded in the instructions that predicts behaviors of the tracked objects;
• and
• plan a trajectory of the vehicle based on tracked objects information from the tracker and predictions from the prediction part.