# US 2019/0244099 A1 · Continual reinforcement learning with a multi-task agent

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2019/0244099 A1
- Document ID: US20190244099A1
- Status: Published application
- Kind: A1
- Application number: 16/268,414
- Filed: 2019-02-05
- Published: 2019-08-08
- Earliest claimed: 2018-02-05
- Assignees: DeepMind Technologies Limited
- Applicants: DeepMind Technologies Limited
- Inventors: Tom Schaul; Matteo Hessel; Hado Philip van Hasselt; Daniel J. Mankowitz
- Web page: https://patentgrep.com/patent/US20190244099A1

## CPC

- G06N 3/006: based on simulated virtual individual or collective life forms, e.g. social simulations or particle swarm optimisation [PSO]
- G06N 3/092: Reinforcement learning
- G05D 1/243: Means capturing signals occurring naturally from the environment, e.g. ambient optical, acoustic, gravitational or magnetic signals (using passive navigation aids external to the vehicle G05D1/244; using signals from positioning sensors located off-board the vehicle G05D1/249)
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU]
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/08: Learning methods
- G06N 3/098: Distributed learning, e.g. federated learning
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G05D: SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES

## Abstract

A method of training an action selection neural network for controlling an agent interacting with an environment to perform different tasks is described. The method includes obtaining a first trajectory of transitions generated while the agent was performing an episode of the first task from multiple tasks; and training the action selection neural network on the first trajectory to adjust the control policies for the multiple tasks. The training includes, for each transition in the first trajectory: generating respective policy outputs for the initial observation in the transition for each task in a subset of tasks that includes the first task and one other task; generating respective target policy outputs for each task using the reward in the transition, and determining an update to the current parameter values based on, for each task, a gradient of a loss between the policy output and the target policy output for the task.

## Claims

### Claim 1

A method of training an action selection neural network used to control an agent interacting with an environment to perform a plurality of different tasks,
wherein the action selection neural network has a plurality of parameters and is configured to:
receive network inputs each comprising (i) a goal signal identifying a task from the plurality of tasks that is being performed by the agent and (ii) an observation characterizing a state of the environment, and
process each network input in accordance with the parameters to generate a respective policy output for each network input that defines a control policy for the agent for performing the task identified by the goal signal, the method comprising:
obtaining a first trajectory of transitions generated while the agent was performing an episode of the first task from the plurality of tasks, each transition in the first trajectory comprising an initial observation characterizing a state of the environment, an action performed by the agent in response to the observation, a reward received as a result of the agent performing the action, and another observation characterizing a subsequent state of the environment; and
training the action selection neural network on the first trajectory to adjust the control policies for the plurality of tasks, comprising, for each transition in the first trajectory:
generating, in accordance with current values of the parameters, respective policy outputs for the initial observation in the transition for each task in a subset of the plurality of tasks that includes the first task and at least one other task;
generating respective target policy outputs for each task in the subset using at least the reward in the transition, and
determining an update to the current values of the parameters based on, for each task in the subset, a gradient of a loss between the policy output for the task and the target policy output for the task.

### Claim 2 (depends on 1)

The method of claim 1, further comprising:
generating the first trajectory of transitions by selecting actions to be performed by the agent while performing the episode of the first task using the action selection neural network and in accordance with the current values of the parameters.

### Claim 3 (depends on 1)

The method of claim 1, further comprising:
selecting, by each of a plurality of actor computing units that each control a respective instance of the agent, a respective task from the plurality of tasks;
generating, by each of the plurality of actor computing units and in parallel, a respective trajectory of transitions by selecting actions to be performed by the agent while performing an episode of the selected task using the action selection neural network in accordance with the current values of the parameters and while the action selection neural network is conditioned on the goal signal for the selected task; and
adding, by each of the plurality of actor computing units, the generated trajectory to a queue of trajectories.

### Claim 4 (depends on 3)

The method of claim 3, further comprising:
generating a batch of trajectories from the trajectories in the queue, wherein the batch includes the first trajectory;
training the action selection neural network on each trajectory in the batch to determine a respective update to the current values of the parameters for each trajectory; and
generating updated values of the parameters from the current values using the updates for the trajectories.

### Claim 5 (depends on 1)

The method of claim 1, wherein generating, in accordance with current values of the parameters, respective policy outputs for the initial observation in the transition for each task in a subset of the plurality of tasks that includes the first task and at least one other task comprises:
processing, for each task in the subset, a network input comprising the initial observation in the transition and the goal signal for the task using the action selection neural network and in accordance with the current values of the parameters.

### Claim 6 (depends on 1)

The method of claim 1, wherein the reward includes a respective pseudo-reward for each task in the subset, and wherein generating respective target policy outputs for each task in the subset using at least the reward in the transition comprises: generating the target policy outputs for each task in the subset using the pseudo-reward for the task and not using the pseudo-rewards for any of the other tasks in the subset.

### Claim 7 (depends on 1)

The method of claim 1, wherein the policy output includes a respective Q value for each action in a set of possible actions that can be performed by the agent, wherein the Q value is an estimate of a return that would be received if the agent performed the action in response to the observation.

### Claim 8 (depends on 7)

The method of claim 7, wherein generating respective target policy outputs for each task in the subset using at least the reward in the transition comprises:
generating respective n-step returns for each task in the subset.

### Claim 9 (depends on 8)

The method of claim 8, wherein generating n-step returns for each task in the subset comprises:
determining whether the performed action in the transition is the action having the highest Q value in the policy output for the task; and
when the performed action in the transition is not the action having the highest Q value, truncating the n-step return using bootstrapping.

### Claim 10 (depends on 1)

The method of claim 1, wherein the action selection neural network is configured to generate an internal goal-independent representation of the state of the environment and to generate the policy output for the task based on the goal signal identifying the task and the goal-independent representation.

### Claim 11

One or more non-transitory computer readable storage media storing instructions that, when executed by one or more computers, cause the one or more computers to perform operations comprising:
training an action selection neural network used to control an agent interacting with an environment to perform a plurality of different tasks,
wherein the action selection neural network has a plurality of parameters and is configured to:
receive network inputs each comprising (i) a goal signal identifying a task from the plurality of tasks that is being performed by the agent and (ii) an observation characterizing a state of the environment, and
process each network input in accordance with the parameters to generate a respective policy output for each network input that defines a control policy for the agent for performing the task identified by the goal signal,
wherein the training comprising:
obtaining a first trajectory of transitions generated while the agent was performing an episode of the first task from the plurality of tasks, each transition in the first trajectory comprising an initial observation characterizing a state of the environment, an action performed by the agent in response to the observation, a reward received as a result of the agent performing the action, and another observation characterizing a subsequent state of the environment; and
training the action selection neural network on the first trajectory to adjust the control policies for the plurality of tasks, comprising, for each transition in the first trajectory:
generating, in accordance with current values of the parameters, respective policy outputs for the initial observation in the transition for each task in a subset of the plurality of tasks that includes the first task and at least one other task;
generating respective target policy outputs for each task in the subset using at least the reward in the transition, and
determining an update to the current values of the parameters based on, for each task in the subset, a gradient of a loss between the policy output for the task and the target policy output for the task.

### Claim 12 (depends on 11)

The one or more non-transitory computer readable storage media of claim 11, wherein the operations further comprises:
generating the first trajectory of transitions by selecting actions to be performed by the agent while performing the episode of the first task using the action selection neural network and in accordance with the current values of the parameters.

### Claim 13 (depends on 11)

The one or more non-transitory computer readable storage media of claim 11, wherein the operations further comprises:
selecting, by each of a plurality of actor computing units that each control a respective instance of the agent, a respective task from the plurality of tasks;
generating, by each of the plurality of actor computing units and in parallel, a respective trajectory of transitions by selecting actions to be performed by the agent while performing an episode of the selected task using the action selection neural network in accordance with the current values of the parameters and while the action selection neural network is conditioned on the goal signal for the selected task; and
adding, by each of the plurality of actor computing units, the generated trajectory to a queue of trajectories.

### Claim 14 (depends on 13)

The one or more non-transitory computer readable storage media of claim 13, wherein the operations further comprises:
generating a batch of trajectories from the trajectories in the queue, wherein the batch includes the first trajectory;
training the action selection neural network on each trajectory in the batch to determine a respective update to the current values of the parameters for each trajectory; and
generating updated values of the parameters from the current values using the updates for the trajectories.

### Claim 15 (depends on 11)

The one or more non-transitory computer readable storage media of claim 11, wherein generating, in accordance with current values of the parameters, respective policy outputs for the initial observation in the transition for each task in a subset of the plurality of tasks that includes the first task and at least one other task comprises:
processing, for each task in the subset, a network input comprising the initial observation in the transition and the goal signal for the task using the action selection neural network and in accordance with the current values of the parameters.

### Claim 16 (depends on 11)

The one or more non-transitory computer readable storage media of claim 11, wherein the reward includes a respective pseudo-reward for each task in the subset, and wherein generating respective target policy outputs for each task in the subset using at least the reward in the transition comprises: generating the target policy outputs for each task in the subset using the pseudo-reward for the task and not using the pseudo-rewards for any of the other tasks in the sub set.

### Claim 17 (depends on 11)

The one or more non-transitory computer readable storage media of claim 11, wherein the policy output includes a respective Q value for each action in a set of possible actions that can be performed by the agent, wherein the Q value is an estimate of a return that would be received if the agent performed the action in response to the observation.

### Claim 18 (depends on 17)

The one or more non-transitory computer readable storage media of claim 17, wherein generating respective target policy outputs for each task in the subset using at least the reward in the transition comprises:
generating respective n-step returns for each task in the subset.

### Claim 19 (depends on 18)

The one or more non-transitory computer readable storage media of claim 18, wherein generating n-step returns for each task in the subset comprises:
determining whether the performed action in the transition is the action having the highest Q value in the policy output for the task; and
when the performed action in the transition is not the action having the highest Q value, truncating the n-step return using bootstrapping.

### Claim 20

A system comprising one or more computers and one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to perform operations comprising:
training an action selection neural network used to control an agent interacting with an environment to perform a plurality of different tasks,
wherein the action selection neural network has a plurality of parameters and is configured to:
receive network inputs each comprising (i) a goal signal identifying a task from the plurality of tasks that is being performed by the agent and (ii) an observation characterizing a state of the environment, and
process each network input in accordance with the parameters to generate a respective policy output for each network input that defines a control policy for the agent for performing the task identified by the goal signal,
wherein the training comprising:
obtaining a first trajectory of transitions generated while the agent was performing an episode of the first task from the plurality of tasks, each transition in the first trajectory comprising an initial observation characterizing a state of the environment, an action performed by the agent in response to the observation, a reward received as a result of the agent performing the action, and another observation characterizing a subsequent state of the environment; and
training the action selection neural network on the first trajectory to adjust the control policies for the plurality of tasks, comprising, for each transition in the first trajectory:
generating, in accordance with current values of the parameters, respective policy outputs for the initial observation in the transition for each task in a subset of the plurality of tasks that includes the first task and at least one other task;
generating respective target policy outputs for each task in the subset using at least the reward in the transition, and
determining an update to the current values of the parameters based on, for each task in the subset, a gradient of a loss between the policy output for the task and the target policy output for the task.