# US 2023/0142461 A1 · Tactical decision-making through reinforcement learning with uncertainty estimation

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2023/0142461 A1
- Document ID: US20230142461A1
- Status: Published application
- Kind: A1
- Application number: 17/996,143
- Filed: 2020-04-20
- Published: 2023-05-11
- Earliest claimed: 2020-04-20
- Assignees: Volvo Autonomous Solutions AB
- Applicants: Volvo Autonomous Solutions AB
- Inventors: Carl-Johan HOEL; Leo LAINE
- Web page: https://patentgrep.com/patent/US20230142461A1

## CPC

- G05D 1/0088: {characterized by the autonomous decision making process, e.g. artificial intelligence, predefined behaviours (using knowledge based models G06N5/00)}
- G06N 3/006: based on simulated virtual individual or collective life forms, e.g. social simulations or particle swarm optimisation [PSO]
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/08: Learning methods
- G06N 3/092: Reinforcement learning
- G06N 7/01: Probabilistic graphical models, e.g. probabilistic networks
- G05D: SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

A method of controlling an autonomous vehicle using a reinforcement learning, RL, agent. The method includes a plurality of training sessions, in which the RL agent interacts with an environment including the autonomous vehicle, each training session having a different initial value and yielding a state-action value function Qk(s, a) dependent on state and action; decision-making, in which the RL agent outputs at least one tentative decision relating to control of the autonomous vehicle; uncertainty estimation on the basis of a variability measure for the plurality of state-action value functions evaluated for a state-action pair corresponding to each of the tentative decisions; and vehicle control, wherein the at least one tentative decision is executed in dependence of the estimated uncertainty.

## Claims

### Claim 1

A method of controlling an autonomous vehicle using a reinforcement learning, RL, agent, the method comprising:
a plurality of training sessions, in which the RL agent interacts with an environment including the autonomous vehicle, each training session having a different initial value and yielding a state-action value function Qk(s, a) dependent on state and action;
decision-making, in which the RL agent outputs at least one tentative decision relating to control of the autonomous vehicle, wherein the decision-making is based on a common state-action value function Q(s, a) obtained by combining the state-action value function Qk(s, a) from the training sessions;
estimating an uncertainty on the basis of a variability measure for the plurality of state-action value functions evaluated for a state-action pair corresponding to each of the tentative decisions; and
vehicle control, wherein the at least one tentative decision is executed in dependence of the estimated uncertainty.

### Claim 2 (depends on 1)

The method of claim 1, wherein each of said at least one tentative decision is executed only if the estimated uncertainty is less than a predefined threshold.

### Claim 3 (depends on 2)

The method of claim 2, wherein:
the decision-making includes the RL agent outputting multiple tentative decisions; and
the vehicle control includes sequential evaluation of the tentative decisions with respect to their estimated uncertainties.

### Claim 4 (depends on 3)

The method of claim 3, wherein a fallback decision is executed if the sequential evaluation does not return a tentative decision to be executed.

### Claim 5 (depends on 1)

The method of claim 1, wherein the decision-making includes tactical decision-making.

### Claim 6 (depends on 1)

The method of claim 1, wherein the RL agent includes at least one neural network.

### Claim 7 (depends on 6)

The method of claim 6, wherein the RL agent is obtained by a policy gradient algorithm, such as an actor-critic algorithm.

### Claim 8 (depends on 6)

The method of claim 6, wherein the RL agent is a Q-learning agent, such as a deep Q network, DQN.

### Claim 9 (depends on 6)

The method of claim 6, wherein the training sessions use an equal number of neural networks.

### Claim 10 (depends on 6)

The method of claim 6, wherein the initial value corresponds to a randomized prior function, RPF.

### Claim 11 (depends on 1)

The method of claim 1, wherein the decision-making is based on a central tendency of said plurality of state-action value functions.

### Claim 12 (depends on 1)

The method of claim 1, wherein the variability measure is one or more of: a variance, a range, a deviation, a variation coefficient, an entropy.

### Claim 13

An arrangement for controlling an autonomous vehicle, comprising:
processing circuitry and memory implementing a reinforcement learning, RL, agent configured to:
interact with an environment including the autonomous vehicle in a plurality of training sessions, each training session having a different initial value and yielding a state-action value function Qk(s, a) dependent on state and action, and
output at least one tentative decision relating to control of the autonomous vehicle, wherein the tentative decision is based on a common state-action value function Q(s, a) obtained by combining the state-action value function Qk(s, a) from the training sessions,
the processing circuitry and memory further implementing an uncertainty estimator configured to estimate an uncertainty on the basis of a variability measure for the plurality of state-action value functions evaluated for a state-action pair corresponding to each of the tentative decisions by the RL agent,
the arrangement further comprising
a vehicle control interface configured to control the autonomous vehicle by executing the at least one tentative decision in dependence of the estimated uncertainty.

### Claim 14 (depends on 13)

A computer program comprising instructions to cause the arrangement of claim 13 to perform the method.

### Claim 15 (depends on 14)

A data carrier carrying the computer program of claim 14.
