Reinforcement learning using distributed prioritized replay
AssigneeDeepMind Technologies Limited
InventorsDavid Budden, Gabriel Barth-Maron, John Quan, Daniel George Horgan
Life of this patent
Estimating the term…
Abstract
Methods, systems, and apparatus, including computer programs encoded on computer storage media, for training an action selection neural network used to select actions to be performed by an agent interacting with an environment. One of the systems includes (i) a plurality of actor computing units, in which each of the actor computing units is configured to maintain a respective replica of the action selection neural network and to perform a plurality of actor operations, and (ii) one or more learner computing units, in which each of the one or more learner computing units is configured to perform a plurality of learner operations.
Claims20
A system for training an action selection neural network having a plurality of network parameters and used to select actions to be performed by an agent interacting with an environment, the system comprising:
- a plurality of actor computing units, each of the actor computing units configured to maintain a respective replica of the action selection neural network and to perform actor operations comprising:
- receiving an observation characterizing a current state of an instance of the environment,
- selecting an action to be performed by the agent using the action selection neural network replica and in accordance with current values of the network parameters,
- obtaining transition data characterizing the environment instance subsequent to the agent performing the selected action,
- generating an experience tuple from the observation, the selected action, and the transition data,
- determining a priority for the experience tuple;
- and
- storing the experience tuple in association with the priority in a shared memory that is accessible to each of the actor computing units;
- and
- one or more learner computing units, wherein each of the one or more learner computing units is configured to perform learner operations comprising:
- sampling a batch of experience tuples from the shared memory, wherein the sampling is biased by the priorities for the experience tuples in the shared memory;
- and
- determining, using the sampled experience tuples, an update to the network parameters using a reinforcement learning technique.
One or more computer readable storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations for training an action selection neural network having a plurality of network parameters and used to select actions to be performed by an agent interacting with an environment, the operations comprising:
- maintaining a plurality of actor computing units, each of the actor computing units configured to maintain a respective replica of the action selection neural network;
- for each of the plurality of actor computing units:
- receiving an observation characterizing a current state of an instance of the environment,
- selecting, using the actor computing unit, an action to be performed by the agent using the action selection neural network replica and in accordance with current values of the network parameters,
- obtaining transition data characterizing the environment instance subsequent to the agent performing the selected action,
- generating an experience tuple from the observation, the selected action, and the transition data,
- determining a priority for the experience tuple, and
- storing the experience tuple in association with the priority in a shared memory that is accessible to each of the plurality of actor computing units;
- maintaining one or more learner computing units;
- for each of the one or more learner computing units:
- sampling, using the learner computing unit, a batch of experience tuples from the shared memory, wherein the sampling is biased by the priorities for the experience tuples in the shared memory;
- and
- determining, using the sampled experience tuples, an update to the network parameters using a reinforcement learning technique.
A method for training an action selection neural network having a plurality of network parameters and used to select actions to be performed by an agent interacting with an environment, the method comprising:
- maintaining a plurality of actor computing units, each of the actor computing units configured to maintain a respective replica of the action selection neural network;
- for each of the plurality of actor computing units:
- receiving an observation characterizing a current state of an instance of the environment,
- selecting, using the actor computing unit, an action to be performed by the agent using the action selection neural network replica and in accordance with current values of the network parameters,
- obtaining transition data characterizing the environment instance subsequent to the agent performing the selected action,
- generating an experience tuple from the observation, the selected action, and the transition data,
- determining a priority for the experience tuple, and
- storing the experience tuple in association with the priority in a shared memory that is accessible to each of the plurality of actor computing units;
- maintaining one or more learner computing units;
- for each of the one or more learner computing units:
- sampling, using the learner computing unit, a batch of experience tuples from the shared memory, wherein the sampling is biased by the priorities for the experience tuples in the shared memory;
- and
- determining, using the sampled experience tuples, an update to the network parameters using a reinforcement learning technique.
Dependencies are inferred from the claim text.