Cross-modality active learning for object detection

A non-transitory computer-readable storage medium comprising at least one program for execution by at least one processor of a first device, the at least one program including instructions which, when executed by the at least one processor, carry out a method comprising:

• generating a first set of predicted bounding boxes based on images from an image sensor and a second set of predicted bounding boxes based on a point cloud from a LiDAR sensor, wherein a respective predicted bounding box of the first set of predicted bounding boxes and the second set of predicted bounding boxes is assigned a classification score indicating a presence of an object class instance within the respective predicted bounding box;
• projecting the first set of predicted bounding boxes and the second set of predicted bounding boxes into a same representation;
• filtering the projections wherein a first subset of predicted bounding boxes satisfying a maximum confidence score is selected from the first set of predicted bounding boxes and a second subset of predicted bounding boxes satisfying the maximum confidence score is selected from the second set of predicted bounding boxes;
• calculating inconsistencies between the first subset of predicted bounding boxes associated with the images from the image sensor and the second subset of predicted bounding boxes associated with the point cloud from the LiDAR sensor based on filtering the projections;
• extracting an informative scene based on the calculated inconsistencies;
• and
• training a first object detection neural network or a second object detection neural network using the informative scene.

A vehicle, comprising:

• at least two sensors comprising an image sensor and a LiDAR sensor, wherein the image sensor is communicatively coupled with a first object detection neural network that generates a first set of predicted bounding boxes based on images from the image sensor and the LiDAR sensor is communicatively coupled with a second object detection neural network that generates a second set of predicted bounding boxes based on a point cloud from the LiDAR sensor, wherein respective predicted bounding box of the first set of predicted bounding boxes and the second set of predicted bounding boxes is assigned a classification score indicating a presence of an object class instance within the respective predicted bounding box;
• at least one computer-readable medium storing computer-executable instructions;
• at least one processor communicatively coupled to the image sensor and the LiDAR sensor and configured to execute the computer-executable instructions, the execution carrying out operations including:
• projecting the first set of predicted bounding boxes and the second set of predicted bounding boxes into a same representation;
• filtering the projections wherein a first subset of predicted bounding boxes satisfying a maximum confidence score is selected from the first set of predicted bounding boxes and a second subset of predicted bounding boxes satisfying the maximum confidence score is selected from the second set of predicted bounding boxes;
• calculating inconsistencies between the first subset of predicted bounding boxes associated with the images from the image sensor and the second subset of predicted bounding boxes associated with the point cloud from the LiDAR sensor based on filtering the projections;
• extracting an informative scene based on the calculated inconsistencies;
• and
• training the first object detection neural network or the second object detection neural network using the informative scene.