Vision-language-planning (vlp) models with agent-wise learning for autonomous driving

A system utilizing a vision-language planning (VLP) machine learning model, the system comprising:

• a camera mounted to a vehicle and configured to generate image data associated with agents in an environment outside the vehicle;
• a processor;
• and
• memory including instructions that, when executed by the processor, cause the processor to:
• process the image data to detect agents in the environment,
• generate a bird eye view (BEV) of the environment based on the image data, wherein the BEV includes BEV features including spatiotemporal information associated with the vehicle and the detected agents,
• input data from the BEV into a perception model, a prediction model, and a planning model of an end-to-end autonomous driving system to generate a predicted trajectory of the vehicle, and
• execute an agent-centric vision-language planning (VLP) machine learning model to:
• (i) extract agent-wise BEV features from the BEV, wherein the agent-wise BEV features are associated with respective agents in the environment,
• (ii) generate natural language text prompts associated with the agents in the environment,
• (iii) extract agent-wise text features from the natural language text prompts, wherein the agent-wise text features are associate with the respective agents in the environment,
• (iv) execute a contrastive learning model to derive similarities between the agent-wise BEV features and the agent-wise text features, and
• (v) refine the BEV features used for the perception model, the prediction model, and the planning model based on the similarities to generate a modified predicted trajectory of the vehicle.

An apparatus for training at least one machine learning model, the apparatus comprising:

• a processor;
• and
• memory including instructions that, when executed by the processor, cause the processor to:
• process image data generated from a camera mounted to a vehicle in order to detect agents in the environment,
• generate a bird eye view (BEV) of the environment based on the image data, wherein the BEV includes BEV features including spatiotemporal information associated with the vehicle and the detected agents,
• input data from the BEV into a perception model, a prediction model, and a planning model of an end-to-end autonomous driving system to generate a predicted trajectory of the vehicle, and
• execute an agent-centric vision-language planning (VLP) machine learning model to:
• (i) extract agent-wise BEV features from the BEV, wherein the agent-wise BEV features are associated with respective agents in the environment,
• (ii) generate natural language text prompts associated with the agents in the environment,
• (iii) extract agent-wise text features from the natural language text prompts, wherein the agent-wise text features are associate with the respective agents in the environment,
• (iv) execute a contrastive learning model to derive similarities between the agent-wise BEV features and the agent-wise text features, and
• (v) refine the BEV features used for the perception model, the prediction model, and the planning model based on the similarities to generate a modified predicted trajectory of the vehicle.