Training action selection neural networks using off-policy actor critic reinforcement learning

A method of training a main neural network having a plurality of main network parameters to determine trained values of the main network parameters by optimizing an objective function, the method comprising:

• receiving a batch of training data comprising a plurality of training examples;
• and
• for each of the plurality of training examples in the batch:
• processing, using the main neural network in accordance with current values of the main network parameters, the training example to generate a main network output for the training example;
• determining a main gradient of the objective function from the main network output;
• processing, using an average neural network having a plurality of average network parameters and in accordance with current values of the average network parameters, the training example to generate an average network output for the training example,
• wherein the current values of the average network parameters represent a running average of values of the main network parameters during the training;
• determining an auxiliary gradient of a measure of a difference between the average network output and the main network output;
• determining a final gradient from the main gradient and the auxiliary gradient;
• and
• determining an update to the current values of the main network parameters from the final gradient.

A system comprising one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the one or more computers to perform operations comprising:

• maintaining a replay memory that stores trajectories generated as a result of interaction of an agent with an environment, each trajectory comprising respective action selection data at each of a plurality of time steps, the action selection data at each time step identifying:
• (i) an observation characterizing a state of the environment,
• (ii) an action performed by the agent in response to the observation,
• (iii) a reward received in response to the agent performing the action, and
• (iv) at least an action selection score assigned to the performed action in determining which action to perform in response to the observation;
• and
• training an action selection neural network having policy parameters on the trajectories in the replay memory, wherein the action selection neural network is configured to:
• receive a provided observation characterizing a state of the environment;
• and
• process the provided observation to generate a network output that defines a score distribution over possible actions that can be performed by the agent in response to the provided observation, and
• wherein training the action selection neural network comprises:
• sampling a trajectory from the replay memory;
• and
• adjusting current values of the policy parameters by training the action selection neural network on the trajectory using an off-policy actor critic reinforcement learning technique, comprising, for each time step in the trajectory from the final time step in the trajectory to the initial time step in the trajectory:
• determining, from the reward for the time step and a current Retrace estimate, a Retrace estimate for the time step;
• and
• determining a main gradient for the time step using at least the Retrace estimate for the time step.

A system comprising:

• one or more computers;
• and
• one or more storage devices communicatively coupled to the one or more computers, wherein the one or more storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to perform operations for training a main neural network having a plurality of main network parameters to determine trained values of the main network parameters by optimizing an objective function, the operations comprising:
• receiving a batch of training data comprising a plurality of training examples;
• and
• for each of the plurality of training examples in the batch:
• processing, using the main neural network in accordance with current values of the main network parameters, the training example to generate a main network output for the training example;
• determining a main gradient of the objective function from the main network output;
• processing, using an average neural network having a plurality of average network parameters and in accordance with current values of the average network parameters, the training example to generate an average network output for the training example,
• wherein the current values of the average network parameters represent a running average of values of the main network parameters during the training;
• determining an auxiliary gradient of a measure of a difference between the average network output and the main network output;
• determining a final gradient from the main gradient and the auxiliary gradient;
• and
• determining an update to the current values of the main network parameters from the final gradient.