# US 2025/0094772 A1 · Training action selection neural networks using off-policy actor critic reinforcement learning and stochastic dueling neural networks

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2025/0094772 A1
- Document ID: US20250094772A1
- Status: Published application
- Kind: A1
- Application number: 18/962,266
- Filed: 2024-11-27
- Published: 2025-03-20
- Earliest claimed: 2016-11-03
- Assignees: DeepMind Technologies Limited
- Applicants: DeepMind Technologies Limited
- Inventors: Ziyu Wang; Nicolas Manfred Otto Heess; Victor Constant Bapst
- Web page: https://patentgrep.com/patent/US20250094772A1

## CPC

- G06N 3/045: Combinations of networks
- G06N 3/006: based on simulated virtual individual or collective life forms, e.g. social simulations or particle swarm optimisation [PSO]
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/0499: Feedforward networks
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06N 3/088: Non-supervised learning, e.g. competitive learning
- G06N 3/09: Supervised learning
- G06N 3/092: Reinforcement learning
- G06N 7/01: Probabilistic graphical models, e.g. probabilistic networks
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for training an action selection neural network. One of the methods includes maintaining a replay memory that stores trajectories generated as a result of interaction of an agent with an environment; and training an action selection neural network having policy parameters on the trajectories in the replay memory, wherein training the action selection neural network comprises: sampling a trajectory from the replay memory; and adjusting current values of the policy parameters by training the action selection neural network on the trajectory using an off-policy actor critic reinforcement learning technique.

## Claims

### Claim at position 1

1-20. (canceled)

### Claim 21

A method performed by one or more computers, the method comprising:
maintaining a replay memory that stores trajectories generated as a result of interaction of an agent with an environment;
training an action selection neural network having policy parameters on the trajectories in the replay memory, wherein training the action selection neural network comprises:
sampling a trajectory from the replay memory; and
adjusting current values of the policy parameters by training the action selection neural network on the trajectory using an off-policy reinforcement learning technique, comprising, for one or more time steps in the trajectory:
determining an importance weight for the time step as a ratio between scores of an action performed at the time step in the trajectory under: (i) an action selection policy defined by the current values of the policy parameters, and (ii) an action selection policy used to control the agent during generation of the trajectory;
generating a truncated importance weight for the time step by truncating the importance weight for the time step to a predefined threshold;
determining a main gradient for the time step with respect to the current values of the policy parameters as a combination of:
a first gradient term that is based on the action performed at the time step in the trajectory and that is scaled by the truncated importance weight; and
a second gradient term that is based on at least one alternate action that is different from the action performed at the time step in the trajectory, wherein the second gradient term compensates for truncation of the importance weight used in the first gradient term; and
updating the current values of the policy parameters using the main gradient for the time step.

### Claim 22 (depends on 21)

The method of claim 21, wherein the second gradient term is based on a plurality of actions from a set of possible actions that can be performed by the agent.

### Claim 23 (depends on 21)

The method of claim 21, wherein the second gradient term is based on all actions from a set of possible actions that can be performed by the agent.

### Claim 24 (depends on 21)

The method of claim 21, wherein the alternate action is stochastically sampled from a set of possible actions that can be performed by the agent.

### Claim 25 (depends on 21)

The method of claim 21, wherein the reinforcement learning technique is an actor critic reinforcement learning technique.

### Claim 26 (depends on 21)

The method of claim 21, wherein the first gradient term is based on a gradient of the score of the action performed at the time step in the trajectory with respect to the current values of the policy parameters.

### Claim 27 (depends on 26)

The method of claim 26, wherein the first gradient term is further based on Retrace estimate for the time step.

### Claim 28 (depends on 27)

The method of claim 27, wherein the first gradient term is based on a product of: (i) the truncated importance weight, (ii) the gradient of the score of the action performed at the time step in the trajectory with respect to the current values of the policy parameters, and (iii) a difference between the Retrace estimate for the time step and a value estimate for the time step.

### Claim 29 (depends on 21)

The method of claim 21, wherein second gradient term is scaled by a compensation weight having a value that depends on a discrepancy between: (i) an alternate importance weight for the time step, and (ii) the predefined threshold.

### Claim 30 (depends on 29)

The method of claim 29, wherein the alternate importance weight for the time step is a ratio between scores of the alternate action under: (i) the action selection policy defined by the current values of the policy parameters, and (ii) the action selection policy used to control the agent during generation of the trajectory.

### Claim 31 (depends on 29)

The method of claim 29, wherein the second gradient term is scaled by a compensation weight that is based on a ratio of the predefined threshold and the alternative importance weight for the time step.

### Claim 32 (depends on 29)

The method of claim 29, wherein second gradient term is based on a gradient of the score of the alternate action with respect to the current values of the policy parameters.

### Claim 33 (depends on 32)

The method of claim 32, wherein the second gradient term is based on a product of: (i) the compensation weight, (ii) the gradient of the score of the alternate action with respect to the current values of the policy parameters, and (iii) a difference between a Q value for the alternate action and a value estimate for the time step.

### Claim 34 (depends on 21)

The method of claim 21, wherein updating the current values of the policy parameters using the main gradient for the time step comprises:
processing an observation for the time step using the action selection neural network to determine a main score distribution;
processing, using an average neural network having a plurality of average network parameters and in accordance with current values of the average network parameters, the observation for the time step to generate an auxiliary score distribution,
wherein the current values of the average network parameters represent a running average of values of the parameters of the action selection network during the training;
determining an auxiliary gradient of a measure of a difference between the auxiliary score distribution and the main score distribution;
determining a final gradient from the main gradient and the auxiliary gradient; and
determining an update to the current values of the policy parameters from the final gradient.

### Claim 35 (depends on 34)

The method of claim 34, wherein the measure of the difference is a Kullback-Leibler divergence.

### Claim 36 (depends on 34)

The method of claim 34, wherein determining the final gradient from the main gradient and the auxiliary gradient comprises:
determining, from the auxiliary gradient and the main gradient, a scaling factor for the main gradient; and
applying the scaling factor to the main gradient to determine the final gradient.

### Claim 37 (depends on 36)

The method of claim 36, wherein the scaling factor s satisfies:
s
=
max
⁢
{
0
,
k
T
⁢
g
-
δ

k

2
2
}
⁢
k
,
where k is the auxiliary gradient, g is the main gradient, and δ is a constant value.

### Claim 38 (depends on 36)

The method of claim 36, wherein applying the scaling factor comprises subtracting the scaling factor from the main gradient.

### Claim 39

A system comprising:
one or more computers; and
one or more storage devices communicatively coupled to the one or more computers, wherein the one or more storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to perform operations comprising:
maintaining a replay memory that stores trajectories generated as a result of interaction of an agent with an environment;
training an action selection neural network having policy parameters on the trajectories in the replay memory, wherein training the action selection neural network comprises:
sampling a trajectory from the replay memory; and
adjusting current values of the policy parameters by training the action selection neural network on the trajectory using an off-policy reinforcement learning technique, comprising, for one or more time steps in the trajectory:
determining an importance weight for the time step as a ratio between scores of an action performed at the time step in the trajectory under: (i) an action selection policy defined by the current values of the policy parameters, and (ii) an action selection policy used to control the agent during generation of the trajectory;
generating a truncated importance weight for the time step by truncating the importance weight for the time step to a predefined threshold;
determining a main gradient for the time step with respect to the current values of the policy parameters as a combination of:
a first gradient term that is based on the action performed at the time step in the trajectory and that is scaled by the truncated importance weight; and
a second gradient term that is based on at least one alternate action that is different from the action performed at the time step in the trajectory, wherein the second gradient term compensates for truncation of the importance weight used in the first gradient term; and
updating the current values of the policy parameters using the main gradient for the time step.

### Claim 40

One or more non-transitory computer storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations comprising:
maintaining a replay memory that stores trajectories generated as a result of interaction of an agent with an environment;
training an action selection neural network having policy parameters on the trajectories in the replay memory, wherein training the action selection neural network comprises:
sampling a trajectory from the replay memory; and
adjusting current values of the policy parameters by training the action selection neural network on the trajectory using an off-policy reinforcement learning technique, comprising, for one or more time steps in the trajectory:
determining an importance weight for the time step as a ratio between scores of an action performed at the time step in the trajectory under: (i) an action selection policy defined by the current values of the policy parameters, and (ii) an action selection policy used to control the agent during generation of the trajectory;
generating a truncated importance weight for the time step by truncating the importance weight for the time step to a predefined threshold;
determining a main gradient for the time step with respect to the current values of the policy parameters as a combination of:
a first gradient term that is based on the action performed at the time step in the trajectory and that is scaled by the truncated importance weight; and
a second gradient term that is based on at least one alternate action that is different from the action performed at the time step in the trajectory, wherein the second gradient term compensates for truncation of the importance weight used in the first gradient term; and
updating the current values of the policy parameters using the main gradient for the time step.
