Continual reinforcement learning with a multi-task agent

One or more non-transitory computer readable storage media storing instructions that, when executed by one or more computers, cause the one or more computers to perform operations comprising:

• training an action selection neural network used to control an agent interacting with an environment to perform a plurality of different tasks,
• wherein the action selection neural network has a plurality of parameters and is configured to:
• receive network inputs each comprising (i) a goal signal identifying a task from the plurality of tasks that is being performed by the agent and (ii) an observation characterizing a state of the environment, and
• process each network input in accordance with the parameters to generate a respective policy output for each network input that defines a control policy for the agent for performing the task identified by the goal signal,
• wherein the training comprising:
• obtaining a first trajectory of transitions generated while the agent was performing an episode of the first task from the plurality of tasks, each transition in the first trajectory comprising an initial observation characterizing a state of the environment, an action performed by the agent in response to the observation, a reward received as a result of the agent performing the action, and another observation characterizing a subsequent state of the environment;
• and
• training the action selection neural network on the first trajectory to adjust the control policies for the plurality of tasks, comprising, for each transition in the first trajectory:
• generating, in accordance with current values of the parameters, respective policy outputs for the initial observation in the transition for each task in a subset of the plurality of tasks that includes the first task and at least one other task;
• generating respective target policy outputs for each task in the subset using at least the reward in the transition, and
• determining an update to the current values of the parameters based on, for each task in the subset, a gradient of a loss between the policy output for the task and the target policy output for the task.

A system comprising one or more computers and one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to perform operations comprising:

• training an action selection neural network used to control an agent interacting with an environment to perform a plurality of different tasks,
• wherein the action selection neural network has a plurality of parameters and is configured to:
• receive network inputs each comprising (i) a goal signal identifying a task from the plurality of tasks that is being performed by the agent and (ii) an observation characterizing a state of the environment, and
• process each network input in accordance with the parameters to generate a respective policy output for each network input that defines a control policy for the agent for performing the task identified by the goal signal,
• wherein the training comprising:
• obtaining a first trajectory of transitions generated while the agent was performing an episode of the first task from the plurality of tasks, each transition in the first trajectory comprising an initial observation characterizing a state of the environment, an action performed by the agent in response to the observation, a reward received as a result of the agent performing the action, and another observation characterizing a subsequent state of the environment;
• and
• training the action selection neural network on the first trajectory to adjust the control policies for the plurality of tasks, comprising, for each transition in the first trajectory:
• generating, in accordance with current values of the parameters, respective policy outputs for the initial observation in the transition for each task in a subset of the plurality of tasks that includes the first task and at least one other task;
• generating respective target policy outputs for each task in the subset using at least the reward in the transition, and
• determining an update to the current values of the parameters based on, for each task in the subset, a gradient of a loss between the policy output for the task and the target policy output for the task.