System and method of cross-modal vision-radar alignment for object-level representation learning

A system, comprising:

• a controller, the controller configured to:
• (i) receive a plurality of paired input images, wherein the paired images includes a first set of images from a first modality and a second set of images from a second modality, wherein the first modality is different than the second modality;
• (ii) output a list of bounding boxes and labels in response to running an image-based object detection model on the first set of images;
• (iii) map each bounding box to a region of interest that is corresponding to the bounding box and associated with the second set of images from the second modality;
• (iv) crop the region of interest from the first set of images and the second set of images to generate a cropped first set of images and a cropped second set of images;
• (v) send the cropped first set of images to a first encoder and the cropped second set of images to a second encoder, wherein the first encoder is configured for the first modality and the second encoder is configured for the second modality;
• (vi) output object-level embeddings for both the cropped first set of images and the cropped second set of images utilizing the first encoder and the second encoder, respectively;
• (vii) identify a loss function associated with the first set of images and the second set of images;
• (viii) in response to when a threshold is not met, repeating steps (i-vii) and when the threshold is met, output final updated parameters associated with the first encoder and the second encoder.

A computer-implemented method for a pre-trained machine-learning network, the computer-implemented method comprising the following steps:

• (i) receiving a plurality of paired input images, wherein the paired images includes a first set of images from a first modality and a second set of images from a second modality, wherein the first modality is different than the second modality;
• (ii) outputting a list of bounding boxes and labels in response to running an image-based object detection model on the first set of images;
• (iii) mapping each bounding box to a region of interest that is corresponding to the bounding box and associated with the second set of images from the second modality;
• (iv) cropping the region of interest from the first set of images and the second set of images to generate a cropped first set of images and a cropped second set of images;
• (v) sending the cropped first set of images to a first encoder and the cropped second set of images to a second encoder, wherein the first encoder is configured for the first modality and the second encoder is configured for the second modality;
• (vi) outputting object-level embeddings for both the cropped first set of images and the cropped second set of images utilizing the first encoder and the second encoder, respectively;
• and
• (vii) identifying a loss function associated with the first set of images and the second set of images.