Method and system for on-the-fly object labeling via cross modality validation in autonomous driving vehicles

A non-transitory, machine-readable medium having data recorded thereon for in-situ perception in an autonomous driving vehicle, the data including instructions to cause a machine to perform operations comprising:

• receiving sensor data acquired continuously by a plurality of sensors deployed on the autonomous driving vehicle, the sensor data providing information about a surrounding region of the autonomous driving vehicle;
• tracking, in accordance with at least one object detection model, one or more items based on a first subset of the sensor data in a first modality and acquired by a first set of one or more sensors from the plurality of sensors, the one or more items appearing in the surrounding region of the autonomous driving vehicle;
• generating validation base data based on a second subset of the sensor data in a second modality and acquired from a second set of one or more sensors from the plurality of sensors;
• labeling, automatically on-the-fly, the one or more items via cross-modality validation based on the validation base data to generate representations of one or more labeled items;
• selecting a first set of events of interest from the one or more labeled items based on a first predetermined configuration;
• sending the first set of events of interest to a global model update unit, which adapts at least one global object detection model based on sets of events of interest from a plurality of autonomous driving vehicles in a fleet, including the first set of events of interest;
• receiving, from the global model update unit, global model update information representing the adapted at least one global object detection model;
• selecting a second set of events of interest from the one or more labeled items based on a second predetermined configuration;
• sending the second set of events of interest to a local model adaptation unit that performs local model adaptation to adjust at least one local object detection model based on the second set of events of interest;
• updating a global object detection model of an in-situ perception system based on the global model update information;
• and
• updating the at least one local object detection model based on information related to one or more updated models received from the global model update unit.

A system for in-situ perception in an autonomous driving vehicle, the system comprising:

• one or more sensor data receivers configured to receive sensor data acquired continuously by a plurality of sensors deployed on the autonomous driving vehicle, the sensor data providing information about a surrounding region of the autonomous driving vehicle;
• and
• a processor, communicably coupled to the one or more sensor data receivers, configured to perform operations comprising:
• tracking, in accordance with at least one object detection model, one or more items based on a first subset of the sensor data in a first modality and acquired by a first set of one or more sensors from the plurality of sensors, the one or more items appearing in the surrounding region of the autonomous driving vehicle;
• generating validation base data based on a second subset of the sensor data in a second modality and acquired from a second set of one or more sensors from the plurality of sensors;
• labeling, automatically on-the-fly, the one or more items via cross-modality validation based on the validation base data to generate representations of one or more labeled items;
• selecting a first set of events of interest from the one or more labeled items based on a first predetermined configuration;
• sending the first set of events of interest to a global model update unit, which adapts at least one global object detection model, to produce an adapted at least one global detection model, based on sets of events of interest from a plurality of autonomous driving vehicles in a fleet, the sets of events of interest including the first set of events of interest;
• receiving, from the global model update unit, global model update information representing the adapted at least one global object detection model;
• selecting a second set of events of interest from the one or more labeled items based on a second predetermined configuration;
• updating a global object detection model of the in-situ perception system based on the received global model update information;
• performing local model adaptation to adjust at least one local object detection model based on the second set of events of interest;
• and
• updating the at least one local object detection model based on information related to one or more updated models received from the global model update unit.