# US 2025/0153736 A1 · Systems and methods for vision-language planning (vlp) foundation models for autonomous driving

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2025/0153736 A1
- Document ID: US20250153736A1
- Status: Published application
- Kind: A1
- Application number: 18/388,606
- Filed: 2023-11-10
- Published: 2025-05-15
- Earliest claimed: 2023-11-10
- Assignees: Robert Bosch GmbH
- Applicants: Robert Bosch GmbH
- Inventors: Chenbin PAN; Burhaneddin YAMAN; Tommaso NESTI; Abhirup MALLIK; Yuliang GUO; Liu REN
- Web page: https://patentgrep.com/patent/US20250153736A1

## CPC

- B60W 60/0015: {specially adapted for safety}
- G06V 10/82: using neural networks
- G06V 20/56: exterior to a vehicle by using sensors mounted on the vehicle
- G06V 20/58: Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
- B60W 2420/403: Image sensing, e.g. optical camera
- B60W: CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

Methods and systems for training an autonomous driving system using a vision-language planning (VLP) model. Image data is obtained from a vehicle-mounted camera, encompassing details about agents situated within the external environment. Via image processing, the system identifies these agents within the environment. A Bird's Eye View (BEV) representation of the surroundings is then generated, encapsulating the spatiotemporal information linked to the vehicle and the recognized agents. Execution of the VLP machine learning model begins by extracting vision-based planning features from the BEV, and receiving or generating textual information characterizing various attributes of the vehicle within the environment. Text-based planning features are extracted from this textual information. To enhance model performance, a contrastive learning model is engaged to establish similarities between the vision-based and text-based planning features, and a predicted trajectory is output based on the similarities.

## Claims

### Claim 1

A method of training an autonomous driving system utilizing a vision-language planning (VLP) machine learning model, the method comprising:
receiving image data generated from a camera mounted to a vehicle, wherein the image data includes agents in an environment outside the vehicle;
via image processing, detecting the agents in the environment based on the image data;
generating a bird eye view (BEV) of the environment based on the image data, wherein the BEV includes spatiotemporal information associated with the vehicle and the detected agents; and
executing a vision-language planning (VLP) machine learning model to:
extract vision-based planning features from the BEV, wherein the vision-based planning features include the spatiotemporal information associated with the vehicle,
generate text information associated with the environment, wherein the text information describes qualities of the vehicle in the environment;
extract text-based planning features from the text information,
execute a contrastive learning model to derive similarities between the vision-based planning features and the text-based planning features, and
generate a predicted trajectory of the vehicle based on the similarities.

### Claim 2 (depends on 1)

The method of claim 1, further comprising:
determining a loss between the predicted trajectory of the vehicle and a ground truth trajectory of the vehicle; and
repeat the steps of claim 1 until convergence to minimize the loss.

### Claim 3 (depends on 1)

The method of claim 1, wherein the text information is generated using a template and ground truth information associated with the environment existent in training data.

### Claim 4 (depends on 1)

The method of claim 1, wherein an output of the contrastive learning is used as a training loss for training the VLP machine learning model.

### Claim 5 (depends on 1)

The method of claim 1, wherein the contrastive learning model includes:
a text encoder configured to output a text-based vector representing text-based features associated with the text information of the environment; and
an image encoder configured to output an image-based vector representing image-based features associated with the detected agents in the BEV.

### Claim 6 (depends on 5)

The method of claim 5, wherein the contrastive learning model is further configured to execute a dot product to evaluate similarities between the text-based vector and the image-based vector.

### Claim 7 (depends on 1)

The method of claim 1, wherein the contrastive learning model is further configured to push apart dissimilarities between the vision-based planning features and the text-based planning features.

### Claim 8 (depends on 1)

The method of claim 1, wherein the agents in the environment include at least one of a pedestrian, another vehicle, or a cyclist.

### Claim 9

A system utilizing a vision-language planning (VLP) machine learning model, the system comprising:
a camera mounted to a vehicle and configured to generate image data associated with agents in an environment outside the vehicle;
a processor; and
memory including instructions that, when executed by the processor, cause the processor to:
process the image data to detect the agents in the environment,
generate a bird eye view (BEV) of the environment based on the image data, wherein the BEV includes spatiotemporal information associated with the vehicle and the detected agents, and
execute a vision-language planning (VLP) machine learning model to:
extract vision-based planning features from the BEV, wherein the vision-based planning features include the at least some of the spatiotemporal information associated with the vehicle,
receive text information associated with the environment, wherein the text information describes qualities of the vehicle in the environment,
extract text-based planning features from the text information,
execute a contrastive learning model to derive similarities between the vision-based planning features and the text-based planning features, and
generate a predicted trajectory of the vehicle based on the similarities.

### Claim 10 (depends on 9)

The system of claim 9, wherein the memory includes further instructions that, when executed by the processor, cause the processor to:
determine a loss between the predicted trajectory of the vehicle and a ground truth trajectory of the vehicle; and
execute the VLP model until convergence to minimize the loss.

### Claim 11 (depends on 9)

The system of claim 9, wherein the text information is generated using a template and ground truth information associated with the environment existent in training data.

### Claim 12 (depends on 9)

The system of claim 9, wherein an output of the contrastive learning is used as a training loss for training the VLP machine learning model.

### Claim 13 (depends on 9)

The system of claim 9, wherein the contrastive learning model includes:
a text encoder configured to output a text-based vector representing text-based features associated with the text information of the environment; and
an image encoder configured to output an image-based vector representing image-based features associated with the detected agents in the BEV.

### Claim 14 (depends on 13)

The system of claim 13, wherein the contrastive learning model is further configured to execute a dot product to evaluate similarities between the text-based vector and the image-based vector.

### Claim 15 (depends on 9)

The system of claim 9, wherein the contrastive learning model is further configured to push apart dissimilarities between the vision-based planning features and the text-based planning features.

### Claim 16 (depends on 9)

The system of claim 9, wherein the agents in the environment include at least one of a pedestrian, another vehicle, or a cyclist.

### Claim 17

A method of training an autonomous driving system, the method comprising:
receiving image data generated from a camera mounted to a vehicle, wherein the image data includes agents in an environment outside the vehicle;
generating a bird eye view (BEV) of the environment based on the image data, wherein the BEV includes spatiotemporal information associated with the vehicle and the agents;
based on the BEV, executing a perception model to detect the agents in the environment and associated information about the detected agents;
based on the BEV, executing a prediction model to estimate trajectories of the detected agents;
based on the BEV, executing a vision-language planning (VLP) model to output a predicted trajectory of the vehicle, wherein the VLP model is configured to:
extract vision-based planning features from the BEV, wherein the vision-based planning features include the spatiotemporal information associated with the vehicle,
receive text information associated with the environment, wherein the text information describes qualities of one or more of the agents in the environment,
extract text-based planning features from the text information,
perform contrastive learning to derive similarities between the vision-based planning features and the text-based planning features, and
output the predicted trajectory based on the similarities.

### Claim 18 (depends on 17)

The method of claim 17, wherein the text information is generated using a template and ground truth information associated with the environment existent in training data.

### Claim 19 (depends on 17)

The method of claim 17, wherein an output of the contrastive learning is used as a training loss for training the VLP machine learning model.

### Claim 20 (depends on 17)

The method of claim 17, wherein the agents in the environment include at least one of a pedestrian, another vehicle, or a cyclist.
