# US 2025/0209337 A1 · Agent control through cultural transmission

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2025/0209337 A1
- Document ID: US20250209337A1
- Status: Published application
- Kind: A1
- Application number: 18/843,608
- Filed: 2023-03-03
- Published: 2025-06-26
- Earliest claimed: 2022-03-03
- Assignees: DeepMind Technologies Limited
- Applicants: DeepMind Technologies Limited
- Inventors: Avishkar Ajay Bhoopchand; Adrian Ashley Collister; Ashley Deloris Edwards; Richard Everett; Edward Fauchon Hughes; Kory Wallace Mathewson; Miruna Pîslar; Alexander Zacherl; Lei Zhang
- Web page: https://patentgrep.com/patent/US20250209337A1

## CPC

- G06N 3/092: Reinforcement learning
- G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU]
- G06N 3/045: Combinations of networks
- G06N 3/096: Transfer learning
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for controlling agents. In particular, an agent can be controlled using a policy neural network that has been trained to allow the agent to achieve cultural transmission after training.

## Claims

### Claim 1

A method performed by one or more computers and for training a policy neural network that is configured to receive a policy input comprising an observation of a state of an environment and to process the policy input to generate a policy output that defines an action to be performed by an agent in response to the observation, the method comprising:
generating a training trajectory by controlling the agent to perform a task episode in the environment across a plurality of time steps using the policy neural network, wherein the environment includes an expert agent that is attempting to perform an instance of a task, and wherein generating the training trajectory comprises, at each of the plurality of time steps:
determining, in accordance with an expert dropout policy for the task episode, whether to cause the expert agent to become unobservable by the agent at the time step;
generating an observation characterizing the environment at the time step, comprising:
in response to determining to cause the expert agent to become unobservable, generating an observation that does not include any sensor measurements of the expert agent;
processing a policy input comprising the observation using the policy neural network to generate a policy output for the observation; and
selecting an action to be performed by the agent at the time step using the policy output; and
training the policy neural network on the training trajectory.

### Claim 2 (depends on 1)

The method of claim 1, wherein generating an observation characterizing the environment at the time step comprises:
in response to determining not to cause the expert agent to become unobservable, generating an observation that includes sensor measurements of the expert agent.

### Claim 3 (depends on 1)

The method of claim 1, wherein the expert dropout policy for the task episode specifies that the expert agent is observable at all of the plurality of time steps.

### Claim 4 (depends on 1)

The method of claim 1, wherein the expert dropout policy for the task episode specifies that the expert agent is unobservable at all of the plurality of time steps.

### Claim 5 (depends on 1)

The method of claim 1, wherein the expert dropout policy for the task episode specifies that the expert agent is only observable at an initial proper subset of the plurality of time steps.

### Claim 6 (depends on 1)

The method of claim 1, wherein the expert dropout policy for the task episode specifies that, for each of the plurality of time steps, the expert agent is observable with a probability p and unobservable with probability 1−p.

### Claim 7 (depends on 1)

The method of claim 1, wherein generating the training trajectory comprises, at one or more of the plurality of time steps:
receiving a respective reward in response to the agent performing the selected action at the time step; and
wherein training the policy neural network on the training trajectory comprises training the policy neural network on the training trajectory using the respective rewards through reinforcement learning.

### Claim 8 (depends on 1)

The method of claim 1, wherein processing a policy input comprising the observation using the policy neural network to generate a policy output for the observation comprises:
processing the policy input using a first subnetwork of the policy neural network to generate a belief representation; and
processing the belief representation using a policy head of the policy neural network to generate the policy output.

### Claim 9 (depends on 8)

The method of claim 8, wherein processing a policy input comprising the observation using the policy neural network to generate a policy output for the observation further comprises:
processing the belief representation using an attention head to generate respective predicted positions of one or more other agents in the environment at a particular time step of the plurality of time steps.

### Claim 10 (depends on 9)

The method of claim 9, wherein the respective prediction positions are ego-centric relative positions relative to the agent.

### Claim 11 (depends on 9)

The method of claim 9, wherein the particular time step is the time step.

### Claim 12 (depends on 9)

The method of claim 9, wherein generating the training trajectory comprises, at one or more of the plurality of time steps:
receiving a respective ground truth position for each of the one or more other agents; and
wherein training the policy neural network on the training trajectory comprises training the policy neural network on the training trajectory to minimize an error between the respective predicted positions and the respective ground truth positions.

### Claim 13 (depends on 1)

The method of any claim 1, wherein the task episode is defined by a respective value for each of a set of task parameters, and wherein generating the training trajectory comprises sampling the respective values for the set of task parameters from a distribution that is parametrized by a set of distribution parameters.

### Claim 14 (depends on 13)

The method of claim 13, further comprising:
determining a cultural transmission metric for the agent; and
adjusting the set of distribution parameters using the cultural transmission metric for the agent.

### Claim 15 (depends on 14)

The method of claim 14, wherein determining the cultural transmission metric for the agent comprises:
generating a plurality of training trajectories, each with a respective expert dropout policy from a plurality of expert dropout policies;
for each training trajectory, obtaining (i) an expert score measuring the performance of the expert agent in performing the instance of the task during the corresponding task episode and (ii) an agent score measuring the performance of the agent in performing the instance of the task during the corresponding task episode; and
determining the cultural transmission metric from the expert scores and the agent scores.

### Claim 16

A method performed by one or more computers and for training a policy neural network that is configured to receive a policy input comprising an observation of a state of an environment and to process the policy input to generate a policy output that defines an action to be performed by an agent in response to the observation, the method comprising:
generating a training trajectory by controlling the agent to perform a task episode in the environment across a sequence of plurality of time steps using the policy neural network, wherein the environment includes an expert agent that is attempting to perform an instance of a task, and wherein generating the training trajectory comprises, at each of the plurality of time steps:
obtaining an observation of the state of the environment at the time step;
processing a policy input comprising the observation using a first subnetwork of the policy neural network to generate a belief representation;
processing the belief representation using a policy head of the policy neural network to generate a policy output for the time step;
processing the belief representation using an attention head of the policy neural network to generate respective predicted positions of one or more other agents in the environment at a time step that is at a particular position relative to the time step in the sequence of time steps; and
selecting an action to be performed by the agent at the time step using the policy output; and
training the policy neural network on the training trajectory, comprising, for one or more of the plurality of time steps:
receiving a respective ground truth position for each of the one or more other agents at the time step that is at a particular position relative to the time step in the sequence of time steps; and
training the policy neural network on the training trajectory to minimize an error between, for each of the one or more time steps, the respective predicted positions and the respective ground truth positions.

### Claim 17 (depends on 16)

The method of claim 16, wherein the respective predicted positions are ego-centric relative positions relative to the agent.

### Claim 18 (depends on 16)

The method of claim 16, wherein the particular position is the same position as the position of the time step.

### Claim 19 (depends on 16)

The method of claim 16, wherein obtaining the observation comprises:
determining, in accordance with an expert dropout policy for the task episode, whether to cause the expert agent to become unobservable by the agent at the time step;
generating the observation characterizing the environment at the time step, comprising:
in response to determining to cause the expert agent to become unobservable, generating an observation that does not include any sensor measurements of the expert agent.

### Claim 20 (depends on 19)

The method of claim 19, wherein generating an observation characterizing the environment at the time step comprises:
in response to determining not to cause the expert agent to become unobservable, generating an observation that include sensor measurements of the expert agent.

### Claim 21 (depends on 19)

The method of claim 19, wherein the expert dropout policy for the task episode specifies that the expert agent is observable at all of the plurality of time steps.

### Claim 22 (depends on 19)

The method of claim 19, wherein the expert dropout policy for the task episode specifies that the expert agent is unobservable at all of the plurality of time steps.

### Claim 23 (depends on 19)

The method of claim 19, wherein the expert dropout policy for the task episode specifies that the expert agent is only observable at an initial proper subset of the plurality of time steps.

### Claim 24 (depends on 19)

The method of claim 19, wherein the expert dropout policy for the task episode specifies that, for each of the plurality of time steps, the expert agent is observable with a probability p and unobservable with probability 1−p.

### Claim 25 (depends on 16)

The method of claim 16, wherein generating the training trajectory comprises, at one or more of the plurality of time steps:
receiving a respective reward in response to the agent performing the selected action at the time step; and
wherein training the policy neural network on the training trajectory comprises training the policy neural network on the training trajectory using the respective rewards through reinforcement learning.

### Claim 26 (depends on 16)

The method of claim 16, wherein the task episode is defined by a respective value for each of a set of task parameters, and wherein generating the training trajectory comprises sampling the respective values for the set of task parameters from a distribution that is parametrized by a set of distribution parameters.

### Claim 27 (depends on 26)

The method of claim 26, further comprising:
determining a cultural transmission metric for the agent; and
adjusting the set of distribution parameters using the cultural transmission metric for the agent.

### Claim 28 (depends on 27)

The method of claim 27, wherein determining the cultural transmission metric for the agent comprises:
generating a plurality of training trajectories, each with a respective expert dropout policy from a plurality of expert dropout policies;
for each training trajectory, obtaining (i) an expert score measuring the performance of the expert agent in performing the instance of the task during the corresponding task episode and (ii) an agent score measuring the performance of the agent in performing the instance of the task during the corresponding task episode; and
determining the cultural transmission metric from the expert scores and the agent scores.

### Claim at position 29

29-40. (canceled)

### Claim 41

A system comprising:
one or more computers; and
one or more storage devices communicatively coupled to the one or more computers, wherein the one or more storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to perform
operations for training a policy neural network that is configured to receive a policy input comprising an observation of a state of an environment and to process the policy input to generate a policy output that defines an action to be performed by an agent in response to the observation, the operations comprising:
generating a training trajectory by controlling the agent to perform a task episode in the environment across a plurality of time steps using the policy neural network, wherein the environment includes an expert agent that is attempting to perform an instance of a task, and wherein generating the training trajectory comprises, at each of the plurality of time steps:
determining, in accordance with an expert dropout policy for the task episode, whether to cause the expert agent to become unobservable by the agent at the time step;
generating an observation characterizing the environment at the time step, comprising:
in response to determining to cause the expert agent to become unobservable, generating an observation that does not include any sensor measurements of the expert agent;
processing a policy input comprising the observation using the policy neural network to generate a policy output for the observation; and
selecting an action to be performed by the agent at the time step using the policy output; and
training the policy neural network on the training trajectory.

### Claim 42

(canceled)
