US 2025/0094772 A1 · TRAINING ACTION SELECTION NEURAL NETWORKS USING OFF-POLICY ACTOR CRITIC REINFORCEMENT LEARNING AND STOCHASTIC DUELING NEURAL NETWORKS

Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for training an action selection neural network. One of the methods includes maintaining a replay memory that stores trajectories generated as a result of interaction of an agent with an environment; and training an action selection neural network having policy parameters on the trajectories in the replay memory, wherein training the action selection neural network comprises: sampling a trajectory from the replay memory; and adjusting current values of the policy parameters by training the action selection neural network on the trajectory using an off-policy actor critic reinforcement learning technique.

Claims21

Dependent claims

1-20. (canceled)

A method performed by one or more computers, the method comprising:

maintaining a replay memory that stores trajectories generated as a result of interaction of an agent with an environment;
training an action selection neural network having policy parameters on the trajectories in the replay memory, wherein training the action selection neural network comprises:
sampling a trajectory from the replay memory;
and
adjusting current values of the policy parameters by training the action selection neural network on the trajectory using an off-policy reinforcement learning technique, comprising, for one or more time steps in the trajectory:
determining an importance weight for the time step as a ratio between scores of an action performed at the time step in the trajectory under:
(i) an action selection policy defined by the current values of the policy parameters, and (ii) an action selection policy used to control the agent during generation of the trajectory;
generating a truncated importance weight for the time step by truncating the importance weight for the time step to a predefined threshold;
determining a main gradient for the time step with respect to the current values of the policy parameters as a combination of:
a first gradient term that is based on the action performed at the time step in the trajectory and that is scaled by the truncated importance weight;
and
a second gradient term that is based on at least one alternate action that is different from the action performed at the time step in the trajectory, wherein the second gradient term compensates for truncation of the importance weight used in the first gradient term;
and
updating the current values of the policy parameters using the main gradient for the time step.

11 more elements

A system comprising:

one or more computers;
and
one or more storage devices communicatively coupled to the one or more computers, wherein the one or more storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to perform operations comprising:
maintaining a replay memory that stores trajectories generated as a result of interaction of an agent with an environment;
training an action selection neural network having policy parameters on the trajectories in the replay memory, wherein training the action selection neural network comprises:
sampling a trajectory from the replay memory;
and
adjusting current values of the policy parameters by training the action selection neural network on the trajectory using an off-policy reinforcement learning technique, comprising, for one or more time steps in the trajectory:
determining an importance weight for the time step as a ratio between scores of an action performed at the time step in the trajectory under:
(i) an action selection policy defined by the current values of the policy parameters, and (ii) an action selection policy used to control the agent during generation of the trajectory;
generating a truncated importance weight for the time step by truncating the importance weight for the time step to a predefined threshold;
determining a main gradient for the time step with respect to the current values of the policy parameters as a combination of:
a first gradient term that is based on the action performed at the time step in the trajectory and that is scaled by the truncated importance weight;
and
a second gradient term that is based on at least one alternate action that is different from the action performed at the time step in the trajectory, wherein the second gradient term compensates for truncation of the importance weight used in the first gradient term;
and
updating the current values of the policy parameters using the main gradient for the time step.

14 more elements

Dependencies are inferred from the claim text.

Training action selection neural networks using off-policy actor critic reinforcement learning and stochastic dueling neural networks

Life of this patent

Training action selection neural networks using off-policy actor critic reinforcement learning and stochastic dueling neural networks

Life of this patent

Abstract

Claims21