Training reinforcement learning agents using augmented temporal difference learning

A system comprising one or more computers and one or more storage devices storing instructions that when executed by one or more computers cause the one or more computers to perform operations for training a Q neural network having a plurality of Q network parameters and used to select actions performed by a reinforcement learning agent interacting with an environment by performing actions that cause the environment to transition states, the method comprising:

• maintaining a replay memory, the replay memory storing a plurality of transitions generated as a result of the reinforcement learning agent interacting with the environment, each transition comprising a respective current observation characterizing a respective current state of the environment, a respective current action performed by the agent in response to the current observation, a respective next observation characterizing a respective next state of the environment, and a reward received in response to the agent performing the current action;
• selecting a plurality of transitions from the replay memory;
• and
• training the Q neural network on the plurality of transitions, comprising, for each transition of the plurality of transitions:
• processing the current observation and the current action in accordance with current values of the Q network parameters to generate an initial Q value for the transition;
• determining, from (i) a current value of a scaling term and (ii) the initial Q value, a scaled Q value for the transition;
• determining, from (i) the current value of the scaling term and (ii) the next observation included in the transition, a scaled temporal difference learning target for the transition;
• determining an error between the scaled temporal difference learning target and the scaled Q value;
• determining, based on computing a gradient of the error with respect to the Q network parameters, an update to the current values of the Q network parameters;
• and
• determining, based on computing a gradient of the error with respect to one or more scaling term parameters, an update to the current value of the scaling term.