Distributional reinforcement learning using quantile function neural networks

A system comprising:

• one or more computers;
• and
• one or more storage devices communicatively coupled to the one or more computers, wherein the one or more storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to perform operations comprising:
• selecting an action to be performed by a reinforcement learning agent interacting with an environment, comprising:
• receiving a current observation characterizing a current state of the environment;
• for each action of a plurality of actions that can be performed by the agent to interact with the environment:
• sampling one or more random values;
• for each random value:
• processing a network input that comprises the action, the current observation, and the random value using an action selection neural network, in accordance with values of a set of action selection neural network parameters, to generate a network output that defines a sample from a return distribution,
• wherein the return distribution defines a probability distribution over possible returns that would result from the agent performing the action in response to the observation;
• and
• determining a measure of central tendency of the samples from the return distribution;
• and
• selecting an action from the plurality of actions to be performed by the agent in response to the current observation using the measures of central tendency for the actions.

One or more non-transitory computer storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations comprising:

• selecting an action to be performed by a reinforcement learning agent interacting with an environment, comprising:
• receiving a current observation characterizing a current state of the environment;
• for each action of a plurality of actions that can be performed by the agent to interact with the environment:
• sampling one or more random values;
• for each random value:
• processing a network input that comprises the action, the current observation, and the random value using an action selection neural network, in accordance with values of a set of action selection neural network parameters, to generate a network output that defines a sample from a return distribution,
• wherein the return distribution defines a probability distribution over possible returns that would result from the agent performing the action in response to the observation;
• and
• determining a measure of central tendency of the samples from the return distribution;
• and
• selecting an action from the plurality of actions to be performed by the agent in response to the current observation using the measures of central tendency for the actions.