Training a policy neural network for controlling an agent using best response policy iteration

A system comprising one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the one or more computers to perform operations comprising:

• training a policy neural network by repeatedly updating the policy neural network at each of a plurality of training iterations, the policy neural network having a plurality of policy network parameters and configured to receive as input a state representation characterizing a state of an environment and to process the state representation in accordance with the policy network parameters to generate a policy output that defines a probability distribution over a set of actions that can be performed by an agent interacting with the environment, comprising, for each of the plurality of training iterations:
• generating training data for the training iteration by controlling the agent in accordance with an improved policy that selects actions in response to input state representations by performing a best response computation, comprising:
• determining a plurality of candidate actions using a candidate policy generated from respective policy neural networks as of one or more preceding iterations;
• determining a value estimate for each candidate action using a candidate value neural network, wherein the candidate value neural network is configured to receive as input a state representation and the candidate action, and to process the state representation and the candidate action to generate a value output that is an estimate of a value of the environment being in the state characterized by the state representation and the agent performing the candidate action to complete a particular task;
• and
• selecting, as the action to be performed by the agent in response to the state representation, the candidate action with a highest value estimate;
• and
• updating the policy neural network by training the policy neural network on the training data.

One or more non-transitory computer-readable storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations comprising:

• training a policy neural network by repeatedly updating the policy neural network at each of a plurality of training iterations, the policy neural network having a plurality of policy network parameters and configured to receive as input a state representation characterizing a state of an environment and to process the state representation in accordance with the policy network parameters to generate a policy output that defines a probability distribution over a set of actions that can be performed by an agent interacting with the environment, comprising, for each of the plurality of training iterations:
• generating training data for the training iteration by controlling the agent in accordance with an improved policy that selects actions in response to input state representations by performing a best response computation, comprising:
• determining a plurality of candidate actions using a candidate policy generated from respective policy neural networks as of one or more preceding iterations;
• determining a value estimate for each candidate action using a candidate value neural network, wherein the candidate value neural network is configured to receive as input a state representation and the candidate action, and to process the state representation and the candidate action to generate a value output that is an estimate of a value of the environment being in the state characterized by the state representation and the agent performing the candidate action to complete a particular task;
• and
• selecting, as the action to be performed by the agent in response to the state representation, the candidate action with a highest value estimate;
• and
• updating the policy neural network by training the policy neural network on the training data.