Enhancing autonomous driving systems with extracted features from bird-eye-view data

A system for enhancing an autonomous driving system utilizing extracted features from bird-eye-view (BEV) data, the system comprising:

• a processor programmed to:
• receive alternative-imaging sensor data associated with an environment about the vehicle, wherein the alternative-imaging sensor data is generated by an alternative-imaging sensor;
• execute a BEV extractor model to extract alternative-imaging BEV feature data based on the received alternative-imaging sensor data;
• generate bird eye view (BEV) based on the extracted alternative-imaging BEV feature data, wherein the BEV includes spatiotemporal information associated with detected agents in the environment;
• execute a BEV enhancement module to:
• (i) extract alternative-imaging BEV features associated with the detected agents in the environment,
• (ii) generate text information associated with the detected agents in the environment,
• (iii) execute a vision-language model (VLM) to extract text-based features from the generated text information, and
• (iv) execute a contrastive learning model to derive a contrastive loss between the alternative-imaging BEV features and the text-based features;
• and
• backpropogate the contrastive loss into the BEV extractor model to enhance the alternative-imaging BEV feature data.

A method of enhancing an autonomous driving system utilizing extracted features from bird-eye-view (BEV) data, the method comprising:

• receiving alternative-imaging sensor data associated with an environment about the vehicle, wherein the alternative-imaging sensor data is generated by an alternative-imaging sensor;
• extracting alternative-imaging BEV feature data based on the received alternative-imaging sensor data;
• generating bird eye view (BEV) based on the extracted alternative-imaging BEV feature data, wherein the BEV includes spatiotemporal information associated with detected agents in the environment;
• extracting alternative-imaging BEV features associated with the detected agents in the environment;
• executing a large language model (LLM) to generate text information associated with the detected agents in the environment;
• executing a vision-language model (VLM) to extract text-based features from the generated text information;
• executing a contrastive learning model to derive a contrastive loss between the alternative-imaging BEV features and the text-based features;
• and
• backpropogating the contrastive loss into the BEV extractor model to enhance the alternative-imaging BEV feature data.