Reinforcement learning using quantile credit assignment

One or more computer-readable storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations for training an action selection neural network to control an agent to take actions in an environment, in response to observations characterizing states of the environment, to perform one or more tasks, the operations comprising:

• maintaining a state action value distribution neural network, wherein the state action value distribution neural network is configured to process an observation at a time step to generate a state action value distribution over estimated returns from a state of the environment represented by the observation and for possible actions of a plurality of possible actions at the time step, wherein the state action value distribution defines a state action value for each of a plurality of quantile levels of the state action value distribution;
• and
• obtaining training data comprising, for each of a plurality of time steps, a tuple defining:
• an observation characterizing a state of an environment at a time step, an action taken by an agent at the time step, a reward received in response to the action;
• and, for each of a plurality of the tuples:
• processing the observation in the tuple representing a state of the environment, using the state action value distribution neural network, to determine the state action value distribution for the observation and for the action in the tuple;
• identifying a quantile level of the state action value distribution that is closest to a return based on the reward in the tuple;
• determining a training target based on at least a difference between the reward in the tuple and a value for the state of the environment determined i) from the state action value distribution at the identified quantile level, or ii) from a value distribution for the observation over estimated returns from the state of the environment, at the identified quantile level;
• training the action selection neural network using the training target, wherein the action selection neural network is configured to process an observation to generate an action selection output for controlling the agent to perform the task.

A system comprising one or more computers and one or more storage devices storing instructions that when executed by one or more computers cause the one or more computers to perform operations for training an action selection neural network to control an agent to take actions in an environment, in response to observations characterizing states of the environment, to perform one or more tasks, the operations comprising:

• maintaining a state action value distribution neural network, wherein the state action value distribution neural network is configured to process an observation at a time step to generate a state action value distribution over estimated returns from a state of the environment represented by the observation and for possible actions of a plurality of possible actions at the time step, wherein the state action value distribution defines a state action value for each of a plurality of quantile levels of the state action value distribution;
• and
• obtaining training data comprising, for each of a plurality of time steps, a tuple defining:
• an observation characterizing a state of an environment at a time step, an action taken by an agent at the time step, a reward received in response to the action;
• and, for each of a plurality of the tuples:
• processing the observation in the tuple representing a state of the environment, using the state action value distribution neural network, to determine the state action value distribution for the observation and for the action in the tuple;
• identifying a quantile level of the state action value distribution that is closest to a return based on the reward in the tuple;
• determining a training target based on at least a difference between the reward in the tuple and a value for the state of the environment determined i) from the state action value distribution at the identified quantile level, or ii) from a value distribution for the observation over estimated returns from the state of the environment, at the identified quantile level;
• training the action selection neural network using the training target, wherein the action selection neural network is configured to process an observation to generate an action selection output for controlling the agent to perform the task.