Training machine learning models on multiple machine learning tasks

A computer-implemented method of training a machine learning model on a plurality of different reinforcement learning tasks,

• wherein the machine learning model has at least a plurality of parameters and has been trained on a first reinforcement learning task using first training data to determine first values of the plurality of parameters of the machine learning model, wherein the first reinforcement learning task comprises controlling a computerized agent to interact with a first simulated environment to achieve a first goal, and
• wherein the method comprises:
• determining, for each of the plurality of parameters, a respective measure of an importance of the parameter to the first reinforcement learning task, comprising:
• computing, based on the first values of the plurality of parameters determined by training the machine learning model on the first reinforcement learning task, an approximation of a posterior distribution over possible values of the plurality of parameters,
• assigning, using the approximation, a value to each of the plurality of parameters, the value being the respective measure of the importance of the parameter to the first reinforcement learning task and approximating a probability that the first value of the parameter after the training on the first reinforcement learning task is a correct value of the parameter given the first training data used to train the machine learning model on the first reinforcement learning task;
• obtaining second training data for training the machine learning model on a second, different reinforcement learning task, wherein the second reinforcement learning task comprises controlling the same computerized agent (i) to interact with a second, different simulated environment to achieve the first goal or a second goal, or (ii) to interact with the first simulated environment to achieve a second goal;
• and
• training the machine learning model on the second reinforcement learning task by training the machine learning model on the second training data to adjust the first values of the plurality of parameters to optimize performance of the machine learning model on the second reinforcement learning task while protecting performance of the machine learning model on the first reinforcement learning task,
• wherein adjusting the first values of the plurality of parameters comprises adjusting the first values of the plurality of parameters to optimize an objective function that depends in part on a penalty term that is based on the determined measures of importance of the plurality of parameters to the first reinforcement learning task.

One or more non-transitory computer storage media encoded with instructions that, when executed by one or more computers, cause the one or more computers to perform operations for training a machine learning model on a plurality of different reinforcement learning tasks,

• wherein the machine learning model has at least a plurality of parameters and has been trained on a first reinforcement learning task using first training data to determine first values of the plurality of parameters of the machine learning model, wherein the first reinforcement learning task comprises controlling a computerized agent to interact with a first simulated environment to achieve a first goal, and
• wherein the operations comprise:
• determining, for each of the plurality of parameters, a respective measure of an importance of the parameter to the first reinforcement learning task, comprising:
• computing, based on the first values of the plurality of parameters determined by training the machine learning model on the first reinforcement learning task, an approximation of a posterior distribution over possible values of the plurality of parameters,
• assigning, using the approximation, a value to each of the plurality of parameters, the value being the respective measure of the importance of the parameter to the first reinforcement learning task and approximating a probability that the first value of the parameter after the training on the first reinforcement learning task is a correct value of the parameter given the first training data used to train the machine learning model on the first reinforcement learning task;
• obtaining second training data for training the machine learning model on a second, different reinforcement learning task, wherein the second reinforcement learning task comprises controlling the same computerized agent (i) to interact with a second, different simulated environment to achieve the first goal or a second goal, or (ii) to interact with the first simulated environment to achieve a second goal;
• and
• training the machine learning model on the second reinforcement learning task by training the machine learning model on the second training data to adjust the first values of the plurality of parameters to optimize performance of the machine learning model on the second reinforcement learning task while protecting performance of the machine learning model on the first reinforcement learning task,
• wherein adjusting the first values of the plurality of parameters comprises adjusting the first values of the plurality of parameters to optimize an objective function that depends in part on a penalty term that is based on the determined measures of importance of the plurality of parameters to the first reinforcement learning task.

A system comprising one or more processors and one or more storage devices storing instructions that, when executed by the one or more processors, cause the one or more processor to perform operations for controlling a computerized agent to interact with a particular simulated environment to achieve a particular goal, the operations comprising:

• receiving an input characterizing a current state of the particular simulated environment;
• processing, using a trained machine learning model, the input to generate an output indicating an action to be performed by the computerized agent to achieve the particular goal;
• and
• causing the computerized agent to perform the action,
• wherein the particular simulated environment is one of a first simulated environment or a second simulated environment and the particular goal is one of a first goal or a second goal,
• wherein the machine learning model has at least a plurality of parameters and has been trained, using a training method, on a first reinforcement learning task using first training data to determine first values of the plurality of parameters of the machine learning model, wherein the first reinforcement learning task comprises controlling the computerized agent to interact with the first simulated environment to achieve a first goal, and
• wherein the training method comprises:
• determining, for each of the plurality of parameters, a respective measure of an importance of the parameter to the first reinforcement learning task, comprising:
• computing, based on the first values of the plurality of parameters determined by training the machine learning model on the first reinforcement learning task, an approximation of a posterior distribution over possible values of the plurality of parameters,
• assigning, using the approximation, a value to each of the plurality of parameters, the value being the respective measure of the importance of the parameter to the first reinforcement learning task and approximating a probability that the first value of the parameter after the training on the first reinforcement learning task is a correct value of the parameter given the first training data used to train the machine learning model on the first reinforcement learning task;
• obtaining second training data for training the machine learning model on a second, different reinforcement learning task, wherein the second reinforcement learning task comprises controlling the same computerized agent (i) to interact with the second, different simulated environment to achieve the first goal or the second goal, or (ii) to interact with the first simulated environment to achieve the second goal;
• and
• training the machine learning model on the second reinforcement learning task by training the machine learning model on the second training data to adjust the first values of the plurality of parameters to optimize performance of the machine learning model on the second reinforcement learning task while protecting performance of the machine learning model on the first reinforcement learning task,
• wherein adjusting the first values of the plurality of parameters comprises adjusting the first values of the plurality of parameters to optimize an objective function that depends in part on a penalty term that is based on the determined measures of importance of the plurality of parameters to the first reinforcement learning task.