Learning options for action selection with meta-gradients in multi-task reinforcement learning

A computer-implemented system for controlling an agent to perform a plurality of tasks while interacting with an environment, wherein the system is configured to, at each of a plurality of time steps, process an input comprising an observation characterizing a current state of the environment to generate an output for selecting an action to be performed by the agent, and receive a task reward in response to the action, the system comprising:

• a manager neural network, and a set of option policy neural networks each for selecting a sequence of actions to be performed by the agent according to a respective option policy;
• wherein the manager neural network is configured to, at a time step:
• process the observation and data identifying one of the tasks currently being performed by the agent, according to parameter values of the manager neural network, to generate an output for selecting a manager action from a set of manager actions, wherein the set of manager actions comprises possible actions that can be performed by the agent and a set of option selection actions, each option selection action selecting one of the option policy neural networks;
• wherein each option policy neural network is configured to, at each of a succession of time steps:
• process the observation for the time step, according to an option policy defined by parameter values of the option policy neural network, to generate an output for selecting an action to be performed by the agent;
• wherein, when the selected manager action is an option selection action, the option policy neural network selected by the manager action generates the output for selecting an action for successive time steps until an option termination criterion is met, and when the selected manager action is one of the possible actions that can be performed by the agent the output for selecting the action is the selected manager action;
• and
• a set of option reward neural networks, one for each respective option policy neural network, each configured to, for a time step:
• process the observation, according to parameter values of the option reward neural network, to generate an option reward for the respective option policy neural network;
• wherein the system is configured to train the set of option reward neural networks and the manager neural network using the task rewards, and to train each of the option policy neural networks using the option reward for the respective option policy neural network.