Training neural networks using a prioritized experience memory

A system comprising one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations for a method for training a neural network used to select actions performed by a reinforcement learning agent interacting with an environment by performing actions that cause the environment to transition states, the method comprising:

• maintaining a replay memory, the replay memory storing a plurality of pieces of experience data that are generated as a result of the reinforcement learning agent interacting with the environment, each piece of experience data having a respective expected learning progress measure;
• selecting a piece of experience data from the replay memory by prioritizing for selection pieces of experience data having relatively higher expected learning progress measures, comprising:
• determining, based on the respective expected learning progress measures for the pieces of experience data, a respective probability for each of the pieces of experience data in the replay memory;
• and
• sampling a piece of experience data from the replay memory in accordance with the determined probabilities;
• training, using a reinforcement learning technique, the neural network on the selected piece of experience data, wherein training the neural network on the selected piece of experience data comprises determining a temporal difference learning error for the selected piece of experience data;
• determining an updated expected learning progress measure for the selected piece of experience data based on an absolute value of the temporal difference learning error;
• and
• associating, in the replay memory, the selected piece of experience data with the updated expected learning progress measure.

A non-transitory computer storage medium encoded with instructions that, when executed by one or more computers, cause the one or more computers to perform operations for a method for training a neural network used to select actions performed by a reinforcement learning agent interacting with an environment by performing actions that cause the environment to transition states, the method comprising:

• maintaining a replay memory, the replay memory storing a plurality of pieces of experience data that are generated as a result of the reinforcement learning agent interacting with the environment, each piece of experience data having a respective expected learning progress measure;
• selecting a piece of experience data from the replay memory by prioritizing for selection pieces of experience data having relatively higher expected learning progress measures, comprising:
• determining, based on the respective expected learning progress measures for the pieces of experience data, a respective probability for each of the pieces of experience data in the replay memory;
• and
• sampling a piece of experience data from the replay memory in accordance with the determined probabilities;
• training, using a reinforcement learning technique, the neural network on the selected piece of experience data, wherein training the neural network on the selected piece of experience data comprises determining a temporal difference learning error for the selected piece of experience data;
• determining an updated expected learning progress measure for the selected piece of experience data based on an absolute value of the temporal difference learning error;
• and
• associating, in the replay memory, the selected piece of experience data with the updated expected learning progress measure.