Method and system for closed loop perception in autonomous driving vehicles

A machine readable and non-transitory medium having data recorded thereon for in-situ perception in an autonomous driving vehicle, wherein the data, once read by the machine, cause the machine to perform the following:

• receiving a plurality of types of sensor data acquired continuously by a plurality of types of sensors deployed on the vehicle, wherein the plurality of types of sensor data provide information about surrounding of the vehicle;
• tracking, in accordance with at least one object detection model, one or more items from a first of the plurality of types of sensor data acquired by one or more of a first type of the plurality of types of sensors, wherein the one or more items appear in the surrounding of the vehicle;
• labeling, automatically on-the-fly, the one or more items via at least one of cross modality validation and cross temporal validation of the one or more items, wherein a label for each of the one or more items is associated with a confidence score;
• adapting locally the at least one object detection model on-the-fly using a first set of events of interest identified from the one or more labeled items based on associated confidence scores;
• sending, via the communication platform, a second set of events of interest, selected from the one or more labeled items, to a model update center, which adapts at least one global object detection model based on second sets of events of interest from a plurality of autonomous driving vehicles in a fleet;
• receiving, from the model update center, model update information, generated via adapting the at least one global object detection model, for updating the at least one object detection model;
• and
• updating the at least one object detection model based on the received model update information.

A system for in-situ perception in an autonomous driving vehicle, comprising:

• one or more sensor data collectors implemented by a processor and configured for receiving a plurality of types of sensor data acquired continuously by a plurality of types of sensors deployed on the vehicle, wherein the plurality of types of sensor data provide information about surrounding of the vehicle;
• an object detection & tracking unit implemented by the processor and configured for tracking, in accordance with at least one object detection model, one or more items from a first of the plurality of types of sensor data acquired by one or more of a first type of the plurality of types of sensors, wherein the one or more items appear in the surrounding of the vehicle;
• an on-the-fly data labeling unit implemented by the processor and configured for
• labeling, automatically on-the-fly, the one or more items via at least one of cross modality validation and cross temporal validation of the one or more items, wherein a label for each of the one or more items is associated with a confidence score,
• adapting locally the at least one object detection model on-the-fly using a first set of events of interest identified from the one or more labeled items based on associated confidence scores;
• a global model update unit implemented by the processor and configured for
• sending, via the communication platform, a set of second set of events of interest, selected from the one or more labeled items, to a model update center, which adapts at least one global object detection model based on second sets of events of interest from a plurality of autonomous driving vehicles in a fleet,
• receiving, from the model update center, model update information, generated via adapting the at least one global object detection model, for updating the at least one object detection model, and
• updating the at least one object detection model based on the received model update information.