# US 2025/0153735 A1 · Techniques for adaptive driving using language models

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2025/0153735 A1
- Document ID: US20250153735A1
- Status: Published application
- Kind: A1
- Application number: 18/933,354
- Filed: 2024-10-31
- Published: 2025-05-15
- Earliest claimed: 2023-11-13
- Assignees: NVIDIA Corporation
- Applicants: Nvidia Corporation
- Inventors: Boyi LI; Boris IVANOVIC; Karen LEUNG; Marco PAVONE; Sushant VEER; Yue WANG
- Web page: https://patentgrep.com/patent/US20250153735A1

## CPC

- B60W 60/0011: {involving control alternatives for a single driving scenario, e.g. planning several paths to avoid obstacles}
- B60W 50/14: Means for informing the driver, warning the driver or prompting a driver intervention
- G06F 40/289: Phrasal analysis, e.g. finite state techniques or chunking
- G06V 10/82: using neural networks
- G06V 20/56: exterior to a vehicle by using sensors mounted on the vehicle
- B60W 2050/143: {Alarm means}
- B60W 2050/146: {Display means}
- B60W 2420/403: Image sensing, e.g. optical camera
- B60W 2555/60: Traffic rules, e.g. speed limits or right of way
- B60W: CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

One embodiment of a method for controlling a vehicle includes receiving first text that includes a description of a scene and a first plan for driving a vehicle, extracting at least one portion of a set of traffic rules based on the description of the scene and the first plan, generating a first prompt that requests driving instructions and includes the description of the scene, the first plan, and the at least one portion of the set of traffic rules, processing the first prompt via a first trained language model to generate a second plan for driving the vehicle, and generating driving instructions based on the second plan.

## Claims

### Claim 1

A computer-implemented method for controlling a vehicle, the method comprising:
receiving first text that includes a description of a scene and a first plan for driving a vehicle;
extracting at least one portion of a set of traffic rules based on the description of the scene and the first plan;
generating a first prompt that requests driving instructions and includes the description of the scene, the first plan, and the at least one portion of the set of traffic rules;
processing the first prompt via a first trained language model to generate a second plan for driving the vehicle; and
generating driving instructions based on the second plan.

### Claim 2 (depends on 1)

The computer-implemented method of claim 1, further comprising receiving second text that indicates a situation, wherein extracting the at least one portion of the set of traffic rules is further based on the situation, and wherein the prompt further includes the situation.

### Claim 3 (depends on 1)

The computer-implemented method of claim 1, wherein extracting the at least one portion of the set of a traffic rules comprises:
generating a second prompt that asks for traffic phrases and includes the description of the scene and the first plan;
processing the second prompt via the first trained language model to extract one or more keywords from the description of the scene and the first plan; and
extracting, from the set of traffic rules, one or more paragraphs that include at least a first keyword that is included in the one or more keywords.

### Claim 4 (depends on 1)

The computer-implemented method of claim 1, wherein the driving instructions are transmitted to a trained planning model, and further comprising:
generating, by the trained planning model and based on the driving instructions, one or more trajectories for the vehicle; and
causing one or more operations to control the vehicle to be performed based on the one or more trajectories.

### Claim 5 (depends on 1)

The computer-implemented method of claim 1, further comprising transmitting the driving instructions to a driver via a speaker device.

### Claim 6 (depends on 1)

The computer-implemented method of claim 1, further comprising displaying the driving instructions to a driver via a display device.

### Claim 7 (depends on 1)

The computer-implemented method of claim 1, further comprising processing image data associated with the vehicle using a vision language model to generate the description of the scene.

### Claim 8 (depends on 1)

The computer-implemented method of claim 1, further comprising generating, via a second trained language model, the first plan based on second text that indicates at least one of sensor data or status information associated with the vehicle.

### Claim 9 (depends on 1)

The computer-implemented method of claim 1, wherein the set of traffic rules are included in a driving handbook.

### Claim 10 (depends on 1)

The computer-implemented method of claim 1, wherein the first trained language model comprises a trained large language model.

### Claim 11

One or more non-transitory computer-readable media storing instructions that, when executed by at least one processor, cause the at least one processor to perform the steps of:
receiving first text that includes a description of a scene and a first plan for driving a vehicle;
extracting at least one portion of a set of traffic rules based on the description of the scene and the first plan;
generating a first prompt that requests driving instructions and includes the description of the scene, the first plan, and the at least one portion of the set of traffic rules;
processing the first prompt via a first trained language model to generate a second plan for driving the vehicle; and
generating driving instructions based on the second plan.

### Claim 12 (depends on 11)

The one or more non-transitory computer-readable media of claim 11, wherein the instructions, when executed by the at least one processor, further cause the at least one processor to perform the step of receiving second text that indicates a situation, wherein extracting the at least one portion of the set of traffic rules is further based on the situation, and wherein the prompt further includes the situation.

### Claim 13 (depends on 11)

The one or more non-transitory computer-readable media of claim 11, wherein extracting the at least one portion of a set of a traffic rules comprises:
generating a second prompt that asks for traffic phrases and includes the description of the scene and the first plan;
processing the second prompt via the first trained language model to extract one or more keywords from the description of the scene and the first plan; and
extracting, from the set of traffic rules, one or more paragraphs that include at least a first keyword that is included in the one or more keywords.

### Claim 14 (depends on 11)

The one or more non-transitory computer-readable media of claim 11, wherein the driving instructions are transmitted to a trained planning model, and the instructions, when executed by the at least one processor, further cause the at least one processor to perform the steps of:
generating, by the trained planning model and based on the driving instructions, one or more trajectories for the vehicle; and
causing one or more operations to control the vehicle to be performed based on the one or more trajectories.

### Claim 15 (depends on 11)

The one or more non-transitory computer-readable media of claim 11, wherein the instructions, when executed by the at least one processor, further cause the at least one processor to perform the step of transmitting the driving instructions to a driver via an audio signal that is output by a speaker device.

### Claim 16 (depends on 11)

The one or more non-transitory computer-readable media of claim 11, wherein the instructions, when executed by the at least one processor, further cause the at least one processor to perform the step of transmitting the driving instructions to a driver via text that is displayed by a display device.

### Claim 17 (depends on 11)

The one or more non-transitory computer-readable media of claim 11, wherein the instructions, when executed by the at least one processor, further cause the at least one processor to perform the step of processing image data associated with the vehicle using a vision language model to generate the description of the scene.

### Claim 18 (depends on 11)

The one or more non-transitory computer-readable media of claim 11, wherein the instructions, when executed by the at least one processor, further cause the at least one processor to perform the step of generating, via a second trained language model, the first plan based on second text that indicates at least one of sensor data or status information associated with the vehicle.

### Claim 19 (depends on 11)

The one or more non-transitory computer-readable media of claim 11, wherein the first text and the first plan are received from at least one of a user or one or more trained machine learning models.

### Claim 20

A system, comprising:
one or more memories storing instructions; and
one or more processors that are coupled to the one or more memories and, when executing the instructions, are configured to:
receive first text that includes a description of a scene and a first plan for driving a vehicle,
extract at least one portion of a set of traffic rules based on the description of the scene and the first plan,
generate a first prompt that requests driving instructions and includes the description of the scene, the first plan, and the at least one portion of the set of traffic rules,
process the first prompt via a first trained language model to generate a second plan for driving the vehicle, and
generate driving instructions based on the second plan.
