# US 2025/0093828 A1 · Training a high-level controller to generate natural language commands for controlling an agent

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2025/0093828 A1
- Document ID: US20250093828A1
- Status: Published application
- Kind: A1
- Application number: 18/892,260
- Filed: 2024-09-20
- Published: 2025-03-20
- Earliest claimed: 2023-09-20
- Assignees: DeepMind Technologies Limited
- Applicants: DeepMind Technologies Limited
- Inventors: Arun Ahuja; Robert David Fergus; Ishita Dasgupta; Kavya Venkata Kota Sai Kopparapu
- Web page: https://patentgrep.com/patent/US20250093828A1

## CPC

- G05B 13/027: {using neural networks only}
- G06N 3/006: based on simulated virtual individual or collective life forms, e.g. social simulations or particle swarm optimisation [PSO]
- G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU]
- G06N 3/09: Supervised learning
- G06N 3/092: Reinforcement learning
- G06F 40/58: Use of machine translation, e.g. for multi-lingual retrieval, for server-side translation for client devices or for real-time translation
- G05B: CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS (systems for controlling or regulating non-electric variables G05D; systems for regulating electric or magnetic variables G05F; control devices or systems insofar as characterised by mechanical features only G05G)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for training a high-level controller neural network for controlling an agent. In particular, the high-level controller neural network generates natural language commands that can be provided as input to a low-level controller neural network, which generates control outputs that can be used to control the agent.

## Claims

### Claim 1

A method performed by one or more computers and for training a high-level controller neural network that is configured to receive an input comprising an observation characterizing a state of an environment being interacted with by an agent and to generate an output defining a natural language command for a low-level controller neural network that generates control outputs for controlling the agent, the method comprising:
obtaining a training data set, the training data set comprising:
a plurality of demonstration trajectories, each demonstration trajectory comprising, for each of a plurality of time steps, a respective observation characterizing a state of a demonstration environment being interacted with by a demonstration agent at the time step and a respective natural language command provided to the demonstration agent at the time step; and
training the high-level controller neural network on the demonstration trajectories in the training data set through supervised learning.

### Claim 2 (depends on 1)

The method of claim 1, wherein training the high-level controller neural network on the demonstration trajectories in the training data set through supervised learning comprises:
training the high-level controller neural network on the demonstration trajectories in the training data set to minimize a behavior cloning loss that measures, for each of the plurality of time steps in each demonstration trajectory, a probability assigned to the respective natural language command provided to the demonstration agent at the time step by an output generated by the high-level controller neural network by processing an input comprising the respective observation for the time step.

### Claim 3 (depends on 1)

The method of claim 1, wherein the training further comprises:
generating a reinforcement learning trajectory, the generating comprising:
at each of a plurality of first time steps in the reinforcement learning trajectory:
receiving a current observation characterizing a state of a training environment being interacted with by a training agent;
processing an input comprising the current observation using the high-level controller neural network to generate an output defining a natural language command for the first time step;
processing an input comprising the natural language command for the first time step using the low-level controller neural network to generate a control output for the first time step;
controlling the training agent using the control output; and
receiving a reward for the first time step; and
training the high-level controller neural network through reinforcement learning using at least the rewards for the first time steps.

### Claim 4 (depends on 3)

The method of claim 3, wherein the input to the low-level controller neural network further comprises the current observation at the first time step.

### Claim 5 (depends on 3)

The method of claim 3, the generating comprising:
at each of a plurality of second time steps in the reinforcement learning trajectory:
receiving a current observation characterizing a state of the training environment being interacted with by the training agent;
determining that criteria are not satisfied for generating a new natural language command; and
in response:
processing an input comprising the natural language command from a most recent first time step using the low-level controller neural network to generate a control output for the second time step;
controlling the training agent using the control output; and
receiving a reward for the second time step.

### Claim 6 (depends on 5)

The method of claim 5, wherein training the high-level controller neural network through reinforcement learning comprises:
training the high-level controller neural network through reinforcement learning using at least the rewards for the first and second time steps.

### Claim 7 (depends on 1)

The method of claim 1, wherein the low-level controller neural network has been pre-trained prior to the training of the high-level controller neural network and is held fixed during the training of the high-level controller neural network.

### Claim 8 (depends on 7)

The method of claim 7, wherein the low-level controller neural network has been pre-trained through supervised learning on a plurality of low-level demonstration trajectories, each low-level demonstration trajectory comprising, for each of a plurality of time steps, a respective observation characterizing a state of the demonstration environment being interacted with by the demonstration agent at the time step, a respective natural language command provided to the demonstration agent at the time step, and an action performed by the demonstration agent at the time step.

### Claim 9 (depends on 8)

The method of claim 8, wherein the low-level controller has been pre-trained through behavior cloning on the plurality of low-level demonstration trajectories.

### Claim 10 (depends on 1)

The method of claim 1, wherein, at each time step, the high-level controller neural network is conditioned on observations at one or more previous time steps.

### Claim 11 (depends on 10)

The method of claim 10, wherein the high-level controller neural network comprises one or more recurrent layers.

### Claim 12 (depends on 10)

The method of claim 10, wherein the high-level controller neural network comprises one or more self-attention layers.

### Claim 13 (depends on 1)

The method of claim 1, wherein the environment is a simulated environment or a real-world environment.

### Claim 14 (depends on 13)

The method of claim 13, wherein the environment is a real-world environment, and the observations are obtained from one or more sensors which sense the real-world environment.

### Claim 15 (depends on 14)

The method of claim 14, wherein the agent is a mechanical robot interacting with the real-world environment.

### Claim 16 (depends on 1)

The method of claim 1, wherein the demonstration environment is the same as the environment.

### Claim 17 (depends on 1)

The method of claim 1, wherein the demonstration environment is different from the environment.

### Claim 18 (depends on 1)

The method of claim 1, wherein the environment is a video game environment and the agent is an agent in the video game environment.

### Claim 19

A system comprising one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the one or more computers to perform operations for training a high-level controller neural network that is configured to receive an input comprising an observation characterizing a state of an environment being interacted with by an agent and to generate an output defining a natural language command for a low-level controller neural network that generates control outputs for controlling the agent, the operations comprising:
obtaining a training data set, the training data set comprising:
a plurality of demonstration trajectories, each demonstration trajectory comprising, for each of a plurality of time steps, a respective observation characterizing a state of a demonstration environment being interacted with by a demonstration agent at the time step and a respective natural language command provided to the demonstration agent at the time step; and
training the high-level controller neural network on the demonstration trajectories in the training data set through supervised learning.

### Claim 20

One or more non-transitory computer-readable media storing instructions that when executed by one or more computers cause the one or more computers to perform operations for training a high-level controller neural network that is configured to receive an input comprising an observation characterizing a state of an environment being interacted with by an agent and to generate an output defining a natural language command for a low-level controller neural network that generates control outputs for controlling the agent, the operations comprising:
obtaining a training data set, the training data set comprising:
a plurality of demonstration trajectories, each demonstration trajectory comprising, for each of a plurality of time steps, a respective observation characterizing a state of a demonstration environment being interacted with by a demonstration agent at the time step and a respective natural language command provided to the demonstration agent at the time step; and
training the high-level controller neural network on the demonstration trajectories in the training data set through supervised learning.
