Agent control through cultural transmission
AssigneeDeepMind Technologies Limited
InventorsAvishkar Ajay Bhoopchand, Adrian Ashley Collister, Ashley Deloris Edwards, Richard Everett, Edward Fauchon Hughes, Kory Wallace Mathewson, Miruna Pîslar, Alexander Zacherl, Lei Zhang
Life of this patent
Estimating the term…
Abstract
Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for controlling agents. In particular, an agent can be controlled using a policy neural network that has been trained to allow the agent to achieve cultural transmission after training.
Claims31
A method performed by one or more computers and for training a policy neural network that is configured to receive a policy input comprising an observation of a state of an environment and to process the policy input to generate a policy output that defines an action to be performed by an agent in response to the observation, the method comprising:
- generating a training trajectory by controlling the agent to perform a task episode in the environment across a plurality of time steps using the policy neural network, wherein the environment includes an expert agent that is attempting to perform an instance of a task, and wherein generating the training trajectory comprises, at each of the plurality of time steps:
- determining, in accordance with an expert dropout policy for the task episode, whether to cause the expert agent to become unobservable by the agent at the time step;
- generating an observation characterizing the environment at the time step, comprising:
- in response to determining to cause the expert agent to become unobservable, generating an observation that does not include any sensor measurements of the expert agent;
- processing a policy input comprising the observation using the policy neural network to generate a policy output for the observation;
- and
- selecting an action to be performed by the agent at the time step using the policy output;
- and
- training the policy neural network on the training trajectory.
A method performed by one or more computers and for training a policy neural network that is configured to receive a policy input comprising an observation of a state of an environment and to process the policy input to generate a policy output that defines an action to be performed by an agent in response to the observation, the method comprising:
- generating a training trajectory by controlling the agent to perform a task episode in the environment across a sequence of plurality of time steps using the policy neural network, wherein the environment includes an expert agent that is attempting to perform an instance of a task, and wherein generating the training trajectory comprises, at each of the plurality of time steps:
- obtaining an observation of the state of the environment at the time step;
- processing a policy input comprising the observation using a first subnetwork of the policy neural network to generate a belief representation;
- processing the belief representation using a policy head of the policy neural network to generate a policy output for the time step;
- processing the belief representation using an attention head of the policy neural network to generate respective predicted positions of one or more other agents in the environment at a time step that is at a particular position relative to the time step in the sequence of time steps;
- and
- selecting an action to be performed by the agent at the time step using the policy output;
- and
- training the policy neural network on the training trajectory, comprising, for one or more of the plurality of time steps:
- receiving a respective ground truth position for each of the one or more other agents at the time step that is at a particular position relative to the time step in the sequence of time steps;
- and
- training the policy neural network on the training trajectory to minimize an error between, for each of the one or more time steps, the respective predicted positions and the respective ground truth positions.
29-40. (canceled)
A system comprising:
- one or more computers;
- and
- one or more storage devices communicatively coupled to the one or more computers, wherein the one or more storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to perform
- operations for training a policy neural network that is configured to receive a policy input comprising an observation of a state of an environment and to process the policy input to generate a policy output that defines an action to be performed by an agent in response to the observation, the operations comprising:
- generating a training trajectory by controlling the agent to perform a task episode in the environment across a plurality of time steps using the policy neural network, wherein the environment includes an expert agent that is attempting to perform an instance of a task, and wherein generating the training trajectory comprises, at each of the plurality of time steps:
- determining, in accordance with an expert dropout policy for the task episode, whether to cause the expert agent to become unobservable by the agent at the time step;
- generating an observation characterizing the environment at the time step, comprising:
- in response to determining to cause the expert agent to become unobservable, generating an observation that does not include any sensor measurements of the expert agent;
- processing a policy input comprising the observation using the policy neural network to generate a policy output for the observation;
- and
- selecting an action to be performed by the agent at the time step using the policy output;
- and
- training the policy neural network on the training trajectory.
(canceled)
Dependencies are inferred from the claim text.