# US 11,027,751 B2 · Reinforcement and model learning for vehicle operation

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,027,751 B2
- Document ID: US11027751B2
- Status: Granted
- Kind: B2
- Application number: 16/757,936
- Filed: 2017-10-31
- Published: 2020-11-05
- Granted: 2021-06-08
- Earliest claimed: 2017-10-31
- Assignees: Nissan North America, Inc.; University of Massachusetts; RENAULT s.a.s.
- Applicants: Nissan North America, Inc.; The University of Massachusetts; Renault S.A.S.
- Inventors: Kyle Hollins Wray; Stefan Witwicki; Shlomo Zilberstein
- Web page: https://patentgrep.com/patent/US11027751B2

## CPC

- B60W 60/001: {Planning or execution of driving tasks}
- B60W 60/0011: {involving control alternatives for a single driving scenario, e.g. planning several paths to avoid obstacles}
- B60W 60/0025: {specially adapted for specific operations}
- G01C 21/3407: {specially adapted for specific applications}
- G05D 1/00: Control of position, course, altitude or attitude of land, water, air or space vehicles, e.g. using automatic pilots (drive control systems specially adapted for autonomous road vehicles B60W60/00)
- G08G 1/166: {for active traffic, e.g. moving vehicles, pedestrians, bikes}
- B60W 2540/00: Input parameters relating to occupants
- B60W 2556/10: Historical data
- B60W: CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
- G01C: MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY (measuring liquid level G01F; radio navigation, determining distance or velocity by use of propagation effects, e.g. Doppler effects, propagation time, of radio waves, analogous arrangements using other waves G01S)
- G05D: SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
- G08G: TRAFFIC CONTROL SYSTEMS (guiding railway traffic, ensuring the safety of railway traffic B61L; arrangement of road signs or traffic signals E01F9/00; radar or analogous systems, sonar systems, lidar systems specially adapted for traffic control G01S13/91, G01S15/88, G01S17/88; {radar or analogous systems, sonar systems, lidar systems specially adapted for anti-collision purposes G01S13/93, G01S15/93, G01S17/93})

## Abstract

Methods and vehicles may be configured to gain experience in the form of state-action and/or action-observation histories for an operational scenario as the vehicle traverses a vehicle transportation network. The histories may be incorporated into a model in the form of learning to improve the model over time. The learning may be used to improve integration with human behavior. Driver feedback may be used in the learning examples to improve future performance and to integrate with human behavior. The learning may be used to create customized scenario solutions. The learning may be used to transfer a learned solution and apply the learned solution to a similar scenario.

## Claims

### Claim 1

A method for use in traversing a vehicle transportation network, the method comprising:
traversing, by an autonomous vehicle, a vehicle transportation network, wherein traversing the vehicle transportation network includes:
determining a route of the autonomous vehicle within the vehicle transportation network;
executing the route of the autonomous vehicle;
detecting an operational scenario based on the route of the autonomous vehicle and a location of the autonomous vehicle;
determining a particular scenario-specific operation control evaluation module based on the operational scenario, wherein the scenario-specific operation control evaluation module includes models that determine a candidate vehicle control action based on an operational environment of the autonomous vehicle, wherein the models include an exploration model based on selecting a low probability action in a semi-random manner and an exploitation model based on selecting a high probability action;
instantiating a scenario-specific operational control evaluation module instance based on the particular scenario-specific operation control evaluation module;
traversing a portion of the vehicle transportation network by executing the candidate vehicle control action using the exploration model or the exploitation model;
observing a state resulting from the execution of the candidate vehicle control action;
updating the scenario-specific operational control evaluation module instance based on the state;
generating a state-action history entry based on the candidate vehicle control action and the state; and
storing the state-action history entry in a scenario-specific operation control database.

### Claim 2 (depends on 1)

The method of claim 1, wherein the exploration model and the exploitation model apply a stored state-action history.

### Claim 3 (depends on 1)

The method of claim 1, wherein the exploitation model is based on selecting the candidate vehicle control action using the particular scenario-specific operation control evaluation module.

### Claim 4 (depends on 1)

The method of claim 1, further comprising:
determining that the operational scenario is not complete; and
determining a second particular scenario-specific operation control evaluation module based on the operational scenario.

### Claim 5 (depends on 1)

The method of claim 1, further comprising:
determining that the operational scenario is complete;
determining driver-initiated vehicle control actions, wherein a number of driver-initiated vehicle control actions is greater than a threshold; and
creating a customized operational scenario based on the observed state and the driver-initiated vehicle control actions.

### Claim 6 (depends on 1)

The method of claim 1, wherein the state is associated with an operational environment.

### Claim 7 (depends on 1)

The method of claim 1, wherein the state is associated with a vehicle state of the autonomous vehicle.

### Claim 8 (depends on 1)

The method of claim 1, wherein the state is associated with a vehicle state of another vehicle in the vehicle transportation network.

### Claim 9 (depends on 1)

The method of claim 1 further comprising:
solving a second operational scenario based on the stored state-action history.

### Claim 10

An autonomous vehicle comprising:
a processor configured to execute instructions stored on a non-transitory computer readable medium to:
determine a route of the autonomous vehicle within a vehicle transportation network;
execute the route of the autonomous vehicle;
detect an operational scenario based on the route of the autonomous vehicle and a location of the autonomous vehicle;
determine a particular scenario-specific operation control evaluation module based on the operational scenario, wherein the scenario-specific operation control evaluation module includes a model that determines a candidate vehicle control action based on an operational environment of the autonomous vehicle, wherein the model is based on a low probability random selection of the candidate vehicle control action;
instantiate a scenario-specific operational control evaluation module instance based on the particular scenario-specific operation control evaluation module;
traverse a portion of the vehicle transportation network based on an execution of the candidate vehicle control action using the model;
observe a state resulting from the execution of the candidate vehicle control action; and
generate a state-action history entry based on the candidate vehicle control action and the state; and
a memory configured to store the state-action history entry in a scenario-specific operation control database.

### Claim 11 (depends on 10)

The autonomous vehicle of claim 10, wherein the model applies the stored state-action history such that a probability of the random selection of the candidate vehicle control action diminishes over time proportional to a volume of the stored state-action history.

### Claim 12 (depends on 10)

The autonomous vehicle of claim 10, wherein the model is based on selecting the candidate vehicle control action using the particular scenario-specific operation control evaluation module.

### Claim 13 (depends on 10)

The autonomous vehicle of claim 10, wherein the processor is further configured to execute instructions stored on the non-transitory computer readable medium to:
determine that the operational scenario is not complete; and
determine a second particular scenario-specific operation control evaluation module based on the operational scenario.

### Claim 14 (depends on 10)

The autonomous vehicle of claim 10, wherein the processor is further configured to execute instructions stored on the non-transitory computer readable medium to:
determine that the operational scenario is complete;
determine driver-initiated vehicle control actions, wherein a number of driver-initiated vehicle control actions is greater than a threshold; and
create a customized operational scenario based on the observed state and the driver-initiated vehicle control actions.

### Claim 15 (depends on 10)

The autonomous vehicle of claim 10, wherein the processor is further configured to execute instructions stored on the non-transitory computer readable medium to:
solve a second operational scenario based on the stored state-action history.