End-to-end trainable advanced driver-assistance systems

A system comprising:

• a memory;
• and
• a processing device communicative coupled to the memory, the processing device configured to:
• generate a set of auto-labeled training data using a first autonomous vehicle (AV) system having multiple sensor modalities, wherein the auto-labeled training data comprises:
• a first set of non-lidar data associated with the first AV system, and
• one or more target predictions for the first set of non-lidar data, said predictions generated based at least on lidar data associated with the first AV system;
• and
• train, using the auto-labeled training data, an end-to-end perception model of a second AV system lacking a lidar sensor to predict presence of one or more objects in a driving environment of the second AV system.

A non-transitory computer-readable storage medium having instructions stored thereon that, when executed by a processing device, cause the processing device to perform operations comprising:

• generating a set of auto-labeled training data using a first autonomous vehicle (AV) system having multiple sensor modalities, wherein the auto-labeled training data comprises:
• a first set of non-lidar data associated with the first AV system, and
• one or more target predictions for the first set of non-lidar data, said predictions generated based at least on lidar data associated with the first AV system;
• and
• training, using the auto-labeled training data, an end-to-end perception model of a second AV system lacking a lidar sensor to predict presence of one or more objects in a driving environment of the second AV system.