Reinforcement learning with adaptive return computation schemes

One or more non-transitory computer storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations for training one or more action selection neural networks to control an agent interacting with an environment to perform episodes of a task, the operations comprising:

• maintaining data specifying a policy for selecting between multiple different return computation schemes, each return computation scheme assigning a different importance to exploring the environment while performing an episode of the task and specifying at least a respective intrinsic reward scaling factor that defines how much an intrinsic reward is scaled before being added to an extrinsic reward that is received from the environment when generating returns, the intrinsic reward scaling factors for two or more of the different return computation schemes being different respective positive numbers;
• causing the agent to perform one or more episodes of the task, comprising, for each episode:
• selecting, using the policy, a return computation scheme from the multiple different return computation schemes;
• controlling the agent to perform the episode of the task to maximize a return computed according to the selected return computation scheme, wherein controlling the agent to perform the episode of the task according to the selected return computation scheme comprises repeatedly performing the following:
• receiving an observation characterizing a current state of the environment;
• processing the observation and data specifying the selected return computation scheme using the one or more action selection neural networks to generate an action selection output;
• and
• selecting an action to be performed by the agent using the action selection output;
• identifying rewards that were generated as a result of the agent performing the episode of the task;
• updating, using the identified rewards for at least one of the one or more episodes of the task, the policy for selecting between multiple different return computation schemes, wherein the policy is updated using a non-stationary multi-armed bandit algorithm having a respective arm corresponding to each of the return computation schemes;
• generating training data from the performing of at least one of the one or more task episodes;
• and
• training the one or more action selection neural networks on the training data through reinforcement learning, wherein the one or more action selection neural networks comprise:
• an intrinsic reward action selection neural network that estimates intrinsic returns computed only from intrinsic rewards generated by an intrinsic reward system based on observations received during interactions with the environment;
• an extrinsic reward action selection neural network that estimates extrinsic returns computed only from extrinsic rewards received from the environment as a result of interactions with the environment;
• and
• wherein training the one or more action selection neural networks on the training data comprises:
• training the intrinsic reward action selection neural network using only intrinsic rewards generated as a result of performances of the task episodes;
• and
• training the extrinsic reward action selection neural network using only extrinsic rewards received during performances of the task episodes.

A system comprising one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the one or more computers to perform operations for training one or more action selection neural networks to control an agent interacting with an environment to perform episodes of a task, the operations comprising:

• maintaining data specifying a policy for selecting between multiple different return computation schemes, each return computation scheme assigning a different importance to exploring the environment while performing an episode of the task and specifying at least a respective intrinsic reward scaling factor that defines how much an intrinsic reward is scaled before being added to an extrinsic reward that is received from the environment when generating returns, the intrinsic reward scaling factors for two or more of the different return computation schemes being different respective positive numbers;
• causing the agent to perform one or more episodes of the task, comprising, for each episode:
• selecting, using the policy, a return computation scheme from the multiple different return computation schemes;
• controlling the agent to perform the episode of the task to maximize a return computed according to the selected return computation scheme, wherein controlling the agent to perform the episode of the task according to the selected return computation scheme comprises repeatedly performing the following:
• receiving an observation characterizing a current state of the environment;
• processing the observation and data specifying the selected return computation scheme using the one or more action selection neural networks to generate an action selection output;
• and
• selecting an action to be performed by the agent using the action selection output;
• identifying rewards that were generated as a result of the agent performing the episode of the task;
• updating, using the identified rewards for at least one of the one or more episodes of the task, the policy for selecting between multiple different return computation schemes, wherein the policy is updated using a non-stationary multi-armed bandit algorithm having a respective arm corresponding to each of the return computation schemes;
• generating training data from the performing of at least one of the one or more task episodes;
• and
• training the one or more action selection neural networks on the training data through reinforcement learning, wherein the one or more action selection neural networks comprise:
• an intrinsic reward action selection neural network that estimates intrinsic returns computed only from intrinsic rewards generated by an intrinsic reward system based on observations received during interactions with the environment;
• an extrinsic reward action selection neural network that estimates extrinsic returns computed only from extrinsic rewards received from the environment as a result of interactions with the environment;
• and
• wherein training the one or more action selection neural networks on the training data comprises:
• training the intrinsic reward action selection neural network using only intrinsic rewards generated as a result of performances of the task episodes;
• and
• training the extrinsic reward action selection neural network using only extrinsic rewards received during performances of the task episodes.