Systems and methods for training a camera-based perception model using machine learning

A system for detecting obstacles and drivable area by an autonomous vehicle in motion, comprising:

• at least one memory;
• at least one hardware processor coupled with the at least one memory and configured, individually or in combination, to:
• input both image data collected from at least one camera and map data into neural network feature extractors configured to output a first set of feature vectors;
• encode, using a transformer encoder, the first set of feature vectors into a second set of Bird's Eye View (BEV) feature vectors, wherein the transformer encoder converts information in the first set of feature vectors from a camera space to a BEV space based on spatial relationships relative to the autonomous vehicle;
• identify, using a detection head, at least one object in the second set of BEV feature vectors;
• generate, using a segmentation head, a BEV segmentation map from the second set of BEV feature vectors, wherein the BEV segmentation map identifies the at least one object and drivable surfaces around the autonomous vehicle;
• compare attributes of the at least one object detected using the detection head and attributes of the at least one object detected using the segmentation head;
• update weights of the segmentation head based on the comparison;
• and
• output, via the updated segmentation head, an updated BEV segmentation map.

A non-transitory computer readable medium storing thereon computer executable instructions for detecting obstacles and drivable area by an autonomous vehicle in motion, including instructions for:

• inputting both image data collected from at least one camera and map data into neural network feature extractors configured to output a first set of feature vectors;
• encoding, using a transformer encoder, the first set of feature vectors into a second set of Bird's Eye View (BEV) feature vectors, wherein the transformer encoder converts information in the first set of feature vectors from a camera space to a BEV space based on spatial relationships relative to the autonomous vehicle;
• identifying, using a detection head, at least one object in the second set of BEV feature vectors;
• generating, using a segmentation head, a BEV segmentation map from the second set of BEV feature vectors, wherein the BEV segmentation map identifies the at least one object and drivable surfaces around the autonomous vehicle;
• comparing attributes of the at least one object detected using the detection head and attributes of the at least one object detected using the segmentation head;
• updating weights of the segmentation head based on the comparison;
• and
• outputting, via the updated segmentation head, an updated BEV segmentation map.