# US 2026/0091792 A1 · Systems and methods for enhancing end-to-end planning for autonomous driving and evaluation in closed-loop environment

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2026/0091792 A1
- Document ID: US20260091792A1
- Status: Published application
- Kind: A1
- Application number: 18/899,781
- Filed: 2024-09-27
- Published: 2026-04-02
- Earliest claimed: 2024-09-27
- Assignees: Robert Bosch GmbH
- Applicants: Robert Bosch GmbH
- Inventors: Feng TAO; Xin YE; Burhaneddin YAMAN; Chenbin PAN; Abhirup MALLIK; Liu Ren
- Web page: https://patentgrep.com/patent/US20260091792A1

## CPC

- B60W 50/06: Improving the dynamic response of the control system, e.g. improving the speed of regulation or avoiding hunting or overshoot
- B60W 50/0097: {Predicting future conditions}
- B60W 60/001: {Planning or execution of driving tasks}
- G06N 3/008: based on physical entities controlled by simulated intelligence so as to replicate intelligent life forms, e.g. based on robots replicating pets or humans in their appearance or behaviour
- G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU]
- G06N 3/045: Combinations of networks
- G06N 3/0455: Auto-encoder networks; Encoder-decoder networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/0475: Generative networks
- G06N 3/08: Learning methods
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06N 3/09: Supervised learning
- G06N 3/092: Reinforcement learning
- G06N 3/094: Adversarial learning
- G06N 3/096: Transfer learning
- G06N 20/00: Machine learning
- G06V 10/774: Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
- G06V 10/806: {of extracted features}
- G06V 10/82: using neural networks
- G06V 20/56: exterior to a vehicle by using sensors mounted on the vehicle
- G06V 20/58: Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
- G06V 20/70: Labelling scene content, e.g. deriving syntactic or semantic representations
- B60W 2050/0088: {Adaptive recalibration}
- B60W 2420/403: Image sensing, e.g. optical camera
- B60W: CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

Methods and systems for training an end-to-end autonomous driving system using a vision-language planning (VLP) machine learning model in a closed-loop environment. Images associated with an environment about a vehicle are generated, and a BEV model is executed to generate a BEV view based on the images. A planning model predicts navigation trajectories based on the BEV. The VLP model enhances the system by extracting vision-based planning features, generating text prompts, and employing a language encoder to create text-based expectation features. A contrastive learning model identifies similarities between vision and text features, boosting the performance of the BEV and planning models. The system undergoes closed-loop evaluation in a simulated environment, capturing metrics to refine the autonomous driving system.

## Claims

### Claim 1

A method of training an end-to-end autonomous driving system utilizing a vision-language planning (VLP) machine learning model in a closed-loop environment, the method comprising:
receiving images generated from a plurality of image sensors mounted to a vehicle;
executing a BEV machine-learning model based on the images to generate a bird eye view (BEV) of the environment;
executing a planning machine-learning model on the BEV to generate predicted trajectories to navigate the autonomous vehicle in the environment;
executing a VLP machine-learning model to improve the end-to-end autonomous driving system, including:
extracting vision-based planning features associated with detected agents within the environment, wherein the vision-based planning features include spatiotemporal information associated with detected agents in the images;
generating text prompts based on the extracted spatiotemporal information associated with detected agents in the images;
passing the text prompts through a language encoder to generate text-based expectation features associated with the detected agents; and
executing a contrastive learning model to derive similarities between the vision-based planning features and the text-based expectation features;
boosting the BEV model and the planning model based on the similarities;
performing closed-loop evaluation of the end-to-end autonomous driving system with the boosted BEV model and the boosted planning model by interacting with a simulated environment in real-time and responding dynamically to actions taken by the vehicle based on the predicted trajectories;
capturing evaluation metrics during the closed-loop evaluation; and
modifying the end-to-end autonomous driving system based on the captured evaluation metrics.

### Claim 2 (depends on 1)

The method of claim 1, wherein the images are generated from an open-source tool that simulates real-world driving environments.

### Claim 3 (depends on 2)

The method of claim 2, wherein the generating of text prompts includes extracting text-based descriptions of ground truth data from the open-source tool.

### Claim 4 (depends on 3)

The method of claim 3, wherein the open-source tool is CARLA.

### Claim 5 (depends on 1)

The method of claim 1, wherein the closed-loop evaluation is performed via a Bench2Drive benchmark.

### Claim 6 (depends on 1)

The method of claim 1, wherein the closed-loop evaluation includes determining a loss between the predicted trajectory of the vehicle and a ground truth trajectory of the vehicle.

### Claim 7 (depends on 1)

The method of claim 1, wherein the text prompts are generated from ground truth labels associated with the images.

### Claim 8 (depends on 7)

The method of claim 7, wherein the contrastive learning model includes:
a text encoder configured to output a text-based vector based on the text prompts; and
an image encoder configured to output an image-based vector representing image-based features associated with the detected agents in the BEV.

### Claim 9

An end-to-end autonomous driving system utilizing a vision-language planning (VLP) machine learning model in a closed-loop environment, the system comprising:
a processor; and
memory including instructions that, when executed by the processor, cause the processor to:
receive image of an environment about a vehicle;
execute a BEV machine-learning model based on the images to generate a bird eye view (BEV) of the environment;
execute a planning machine-learning model on the BEV to generate predicted trajectories to navigate an autonomous vehicle in the environment;
execute a VLP machine-learning model to improve the end-to-end autonomous driving system by:
extracting vision-based planning features associated with detected agents within the environment, wherein the vision-based planning features include spatiotemporal information associated with detected agents in the images;
generating text prompts based on the extracted spatiotemporal information associated with detected agents in the images;
passing the text prompts through a language encoder to generate text-based expectation features associated with the detected agents; and
executing a contrastive learning model to derive similarities between the vision-based planning features and the text-based expectation features;
boost the BEV model and the planning model based on the similarities; and
perform closed-loop evaluation of the end-to-end autonomous driving system with the boosted BEV model and the boosted planning model by interacting with a simulated environment in real-time and responding dynamically to actions taken by the vehicle based on the predicted trajectories.

### Claim 10 (depends on 9)

The system of claim 9, wherein the memory includes further instructions that, when executed by the processor, cause the processor to:
capture evaluation metrics during the closed-loop evaluation; and
modify the end-to-end autonomous driving system based on the captured evaluation metrics.

### Claim 11 (depends on 9)

The system of claim 9, wherein the images are generated from an open-source tool that simulates real-world driving environments.

### Claim 12 (depends on 11)

The system of claim 11, wherein the generating of text prompts includes extracting text-based descriptions of ground truth data from the open-source tool.

### Claim 13 (depends on 12)

The system of claim 12, wherein the open-source tool is CARLA.

### Claim 14 (depends on 9)

The system of claim 9, wherein the closed-loop evaluation is performed via a Bench2Drive benchmark.

### Claim 15 (depends on 9)

The system of claim 9, wherein the closed-loop evaluation includes determining a loss between the predicted trajectory of the vehicle and a ground truth trajectory of the vehicle.

### Claim 16 (depends on 9)

The system of claim 9, wherein the text prompts are generated from ground truth labels associated with the images.

### Claim 17 (depends on 7)

The system of claim 7, wherein the contrastive learning model includes:
a text encoder configured to output a text-based vector based on the text prompts; and
an image encoder configured to output an image-based vector representing image-based features associated with the detected agents in the BEV.

### Claim 18

A method comprising:
receiving images associated with an environment about a vehicle;
executing a BEV model on the images to generate a bird eye view (BEV) of the environment;
executing a planning model on the BEV to generate predicted trajectories to navigate an autonomous vehicle in the environment;
executing a VLP model to improve the end-to-end autonomous driving system by:
extracting vision-based planning features associated with agents within the environment, wherein the vision-based planning features include spatiotemporal information associated with agents;
generating text prompts based on the extracted spatiotemporal information;
passing the text prompts through a language encoder to generate text-based expectation features; and
executing a contrastive learning model to derive similarities between the vision-based planning features and the text-based expectation features;
boosting the BEV model and the planning model based on the similarities; and
performing closed-loop evaluation of the end-to-end autonomous driving system with the boosted BEV model and the boosted planning model by interacting with a simulated environment in real-time and responding dynamically to actions taken by the vehicle based on the predicted trajectories.

### Claim 19 (depends on 18)

The method of claim 18, further comprising:
capturing evaluation metrics during the closed-loop evaluation; and
modifying the end-to-end autonomous driving system based on the captured evaluation metrics.

### Claim 20 (depends on 18)

The method of claim 18, wherein the images are generated from CARLA that simulates real-world driving environments, and wherein the generating of text prompts includes extracting text-based descriptions of ground truth data from CARLA.
