Reinforcement learning with scheduled auxiliary control

A system comprising one or more computers and one or more storage devices storing instructions that when executed by one or more computers cause the one or more computers to perform operations of training a primary policy neural network used to select actions performed by an agent interacting with an environment to cause the agent to perform a primary task, the operations comprising:

• maintaining data specifying parameter values for the primary policy neural network and one or more auxiliary policy neural networks, wherein each auxiliary policy neural network is configured to select actions to be performed by the agent to cause the agent to perform a respective auxiliary task that is different from the primary task;
• maintaining data specifying, for each of a plurality of tasks that includes the primary task and each of the one or more auxiliary tasks, a respective reward estimate, wherein the respective reward estimate for each of the auxiliary tasks is an expected reward for the primary task that would be received after the auxiliary task is selected as a current task at a particular selection time step in a particular training episode given a sequence of one or more tasks that were selected at respective earlier selection time steps in the particular training episode;
• controlling the agent during a training episode comprising a plurality of time steps,
• the controlling comprising, at each of a plurality of selection time steps during the training episode:
• receiving an observation characterizing a current state of the environment at the selection time step,
• selecting, from the primary task and the one or more auxiliary tasks, a current task for the selection time step using a learned task scheduling policy that is applied to the respective reward estimates for the plurality of tasks in the maintained data,
• identifying, from the primary policy neural network and the one or more auxiliary policy neural networks, the policy neural network that corresponds to the selected current task;
• processing an input comprising the observation using the identified policy neural network corresponding to the selected current task to select an action to be performed by the agent in response to the observation,
• causing the agent to perform the selected action,
• in response to the agent performing the selected action, obtaining a respective reward for each of the tasks,
• generating an experience tuple comprising data identifying the observation, the selected action, and the respective rewards for the each of the tasks, and
• adding the experience tuple to training data for the primary policy neural network and the auxiliary policy neural networks;
• updating the learned task scheduling policy using the rewards obtained during the training episode comprising, for each auxiliary task that was selected during the training episode, updating the maintained data for the auxiliary task based on rewards that were obtained for the primary task after the auxiliary task was selected during the task episode;
• sampling an experience tuple from the training data;
• and
• training each of the policy neural networks using the sampled experience tuple, comprising, for each policy neural network, training the policy neural network using the reward for the corresponding task in the sampled experience tuple.

One or more non-transitory computer-readable storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations of training a primary policy neural network used to select actions performed by an agent interacting with an environment to cause the agent to perform a primary task, the operations comprising:

• maintaining data specifying parameter values for the primary policy neural network and one or more auxiliary policy neural networks, wherein each auxiliary policy neural network is configured to select actions to be performed by the agent to cause the agent to perform a respective auxiliary task that is different from the primary task;
• maintaining data specifying, for each of a plurality of tasks that includes the primary task and each of the one or more auxiliary tasks, a respective reward estimate, wherein the respective reward estimate for each of the auxiliary tasks is an expected reward for the primary task that would be received after the auxiliary task is selected as a current task at a particular selection time step in a particular training episode given a sequence of one or more tasks that were selected at respective earlier selection time steps in the particular training episode;
• controlling the agent during a training episode comprising a plurality of time steps,
• the controlling comprising, at each of a plurality of selection time steps during the training episode:
• receiving an observation characterizing a current state of the environment at the selection time step,
• selecting, from the primary task and the one or more auxiliary tasks, a current task for the selection time step using a learned task scheduling policy that is applied to the respective reward estimates for the plurality of tasks in the maintained data,
• identifying, from the primary policy neural network and the one or more auxiliary policy neural networks, the policy neural network that corresponds to the selected current task;
• processing an input comprising the observation using the identified policy neural network corresponding to the selected current task to select an action to be performed by the agent in response to the observation,
• causing the agent to perform the selected action,
• in response to the agent performing the selected action, obtaining a respective reward for each of the tasks,
• generating an experience tuple comprising data identifying the observation, the selected action, and the respective rewards for the each of the tasks, and
• adding the experience tuple to training data for the primary policy neural network and the auxiliary policy neural networks;
• updating the learned task scheduling policy using the rewards obtained during the training episode comprising, for each auxiliary task that was selected during the training episode, updating the maintained data for the auxiliary task based on rewards that were obtained for the primary task after the auxiliary task was selected during the task episode;
• sampling an experience tuple from the training data;
• and
• training each of the policy neural networks using the sampled experience tuple, comprising, for each policy neural network, training the policy neural network using the reward for the corresponding task in the sampled experience tuple.