Machine learning systems with counterfactual interventions
AssigneeDeepMind Technologies Limited
InventorsNeil Charles Rabinowitz, Nicholas Andrew Roy, Junkyung Kim
Life of this patent
Estimating the term…
Abstract
Systems, methods, and computer program, for training and using a machine learning system to control an agent to perform a task. The machine learning system is trained using counterfactual internal states so that it can provide an output that explains the behavior of the system in causal terms e.g. in terms of aspects of its environment that cause the system to select particular actions for the agent.
Claims21
A computer-implemented method of training a machine learning system configured to monitor the control of an agent in an environment performing a task, wherein the machine learning system is configured to, for each of a succession of time steps:
- obtain an observation characterizing a state of the environment for a current time step;
- process an internal state of the machine learning system at the current time step using an action selection subsystem of the machine learning system to generate an action selection policy output, wherein the internal state of the machine learning system at the current time step depends on the observation for the current time step;
- process the internal state of the machine learning system at the current time step using a decoder neural network to generate a decoder output for the current time step, wherein the decoder output for the current time step describes the internal state of the machine learning system at the current time step;
- select an action to be performed by the agent at the current time step in response to the observation using the action selection policy output;
- cause the agent to perform the selected action;
- and
- provide a signal derived from the decoder output for the current time step as a monitoring signal for monitoring the control of the agent in the environment;
- the method comprising training the machine learning system by:
- determining the internal state of the machine learning system at a next time step by processing, using an internal state updating subsystem, a combination of the decoder output for the current time step, the observation for the next time step, and a counterfactual internal state, wherein the counterfactual internal state is different to the internal state of the machine learning system at the current time step.
-25. (canceled)
A system comprising one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations for training a machine learning system configured to monitor the control of an agent in an environment performing a task, wherein the machine learning system is configured to, for each of a succession of time steps:
- obtain an observation characterizing a state of the environment for a current time step;
- process an internal state of the machine learning system at the current time step using an action selection subsystem of the machine learning system to generate an action selection policy output, wherein the internal state of the machine learning system at the current time step depends on the observation for the current time step;
- process the internal state of the machine learning system at the current time step using a decoder neural network to generate a decoder output for the current time step, wherein the decoder output for the current time step describes the internal state of the machine learning system at the current time step;
- select an action to be performed by the agent at the current time step in response to the observation using the action selection policy output;
- cause the agent to perform the selected action;
- and
- provide a signal derived from the decoder output for the current time step as a monitoring signal for monitoring the control of the agent in the environment;
- the operations for training the machine learning system comprising:
- determining the internal state of the machine learning system at a next time step by processing, using an internal state updating subsystem, a combination of the decoder output for the current time step, the observation for the next time step, and a counterfactual internal state, wherein the counterfactual internal state is different to the internal state of the machine learning system at the current time step.
A computer storage medium encoded with instructions that, when executed by one or more computers, cause the one or more computers to perform operations for training a machine learning system configured to monitor the control of an agent in an environment performing a task. wherein the machine learning system is configured to, for each of a succession of time steps:
- obtain an observation characterizing a state of the environment for a current time step;
- process an internal state of the machine learning system at the current time step using an action selection subsystem of the machine learning system to generate an action selection policy output, wherein the internal state of the machine learning system at the current time step depends on the observation for the current time step;
- process the internal state of the machine learning system at the current time step using a decoder neural network to generate a decoder output for the current time step, wherein the decoder output for the current time step describes the internal state of the machine learning system at the current time step;
- select an action to be performed by the agent at the current time step in response to the observation using the action selection policy output;
- cause the agent to perform the selected action;
- and
- provide a signal derived from the decoder output for the current time step as a monitoring signal for monitoring the control of the agent in the environment;
- the operations for training the machine learning system comprising:
- determining the internal state of the machine learning system at a next time step by processing, using an internal state updating subsystem, a combination of the decoder output for the current time step, the observation for the next time step, and a counterfactual internal state, wherein the counterfactual internal state is different to the internal state of the machine learning system at the current time step.
Dependencies are inferred from the claim text.