# US 2026/0310977 A1 · Agent control using token-based dynamics models

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2026/0310977 A1
- Document ID: US20260310977A1
- Status: Published application
- Kind: A1
- Application number: 19/481,609
- Filed: 2024-05-03
- Published: 2026-10-08
- Earliest claimed: 2023-05-05
- Assignees: GDM Holding LLC
- Applicants: GDM Holding LLC
- Inventors: Jingwei Zhang; Emilio Parisotto; Jost Tobias Springenberg; Leonard Hasenclever; Nicolas Manfred Otto Heess; Arunkumar Byravan; Sarah Maria Elisabeth Bechtle; Ingmar Fabian Fabian Schubert
- Web page: https://patentgrep.com/patent/US20260310977A1

## CPC

- G06N 3/09: Supervised learning
- G06N 3/045: Combinations of networks
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for controlling agents using sequence-processing neural networks. In particular, the sequence-processing neural network is used as a dynamics model of the environment in order to perform planning when selecting actions to be performed by an agent.

## Claims

### Claim 1

A method performed by one or more computers and for controlling an agent interacting with an environment to perform a task episode, the method comprising:
at each of a plurality of current time steps in the task episode:
obtaining an observation characterizing a current state of the environment at the time step;
generating a plurality of planned trajectories starting from the current state of the environment, wherein each planned trajectory comprises data representing a respective observation at each of a sequence of planning time steps that starts at the current time step and includes a plurality of future time steps that are each after the current time step, the generating comprising, for each planned trajectory and at each planning time step other than a last planning time step in the sequence:
identifying an action to be performed by the agent in response to the observation at the planning time step in the planned trajectory; and
generating data specifying the observation at the subsequent planning time step in the planned trajectory, comprising processing a first input sequence for the planning time step that comprises a sequence of tokens from a vocabulary of tokens and representing the respective observation in the planned trajectory at the planning time step and one or more tokens from the vocabulary of tokens and representing the identified action at the planning time step, using a sequence-processing neural network, to generate an output sequence that comprises a sequence of tokens from the vocabulary of tokens that represents the observation at the subsequent planning time step in the planned trajectory;
selecting a planned trajectory from the plurality of planned trajectories; and
causing the agent to perform the identified action for the first planning time step in the selected planned trajectory.

### Claim 2 (depends on 1)

The method of claim 1, wherein the sequence-processing neural network is an auto-regressive neural network.

### Claim 3 (depends on 1)

The method of claim 1, wherein selecting a planned trajectory from the plurality of planned trajectories comprises:
determining a respective quality score for each planned trajectory; and
selecting the planned trajectory having a highest quality score.

### Claim 4 (depends on 3)

The method of claim 3, wherein determining a respective quality score for each planned trajectory comprises:
determining a respective reward for each planning time step after the first planning time step in the planned trajectory; and
determining the respective quality score for the planned trajectory from the respective rewards.

### Claim 5 (depends on 4)

The method of claim 4, wherein determining a respective reward for each planning time step after the first planning time step in the planned trajectory comprises:
determining the respective reward for the planning time step from the sequence of tokens representing the observation at the planning time step in the planned trajectory.

### Claim 6 (depends on 4)

The method of claim 4, wherein determining a respective reward for each planning time step after the first planning time step in the planned trajectory comprises:
processing a second input sequence for the planning time step comprising the sequence of tokens representing the observation at the planning time step in the planned trajectory, using the sequence-processing neural network, to generate a token from the vocabulary that represents the respective reward for the planning time step.

### Claim 7 (depends on 6)

The method of claim 6, wherein the second sequence comprises, for each planning time step preceding the planning time step in the planned trajectory, the sequence of tokens representing the observation at the preceding planning time step and the identified action for the preceding planning time step.

### Claim 8 (depends on 7)

The method of claim 7, wherein the second sequence further comprises, for each planning time step preceding the planning time step in the planned trajectory, the token that represents the respective reward for the planning time step.

### Claim 9 (depends on 1)

The method of claim 1, wherein, for each planning time step, the first input sequence further comprises, for each planning time step preceding the planning time step in the planned trajectory, the sequence of tokens representing the observation at the preceding planning time step and the identified action for the preceding planning time step.

### Claim 10 (depends on 9)

The method of claim 9, wherein, for each planning time step, the first input sequence comprises a historical sequence that comprises, for each of one or more time steps preceding the current time step in the task episode, a sequence of tokens from the vocabulary representing the observation at the time step and one or more tokens from the vocabulary representing an action performed by the agent at the time step.

### Claim 11 (depends on 1)

The method of claim 1, wherein identifying an action to be performed by the agent in response to the observation at the planning time step in the planned trajectory comprises:
randomly selecting the action to be performed at the planning time step in the planned trajectory from a set of actions that can be represented by one or more tokens from the vocabulary.

### Claim 12 (depends on 1)

The method of claim 1, wherein identifying an action to be performed by the agent in response to the observation at the planning time step in the planned trajectory comprises:
applying a control policy for the agent to at least the observation at the planning time step in the planned trajectory to select the action to be performed by the agent.

### Claim 13 (depends on 12)

The method of claim 12, wherein applying a control policy for the agent to at least the observation at the planning time step in the planned trajectory to select the action to be performed by the agent comprises:
processing a third input sequence for the planning time step comprising the sequence of tokens representing the observation at the planning time step in the planned trajectory using the sequence-processing neural network to generate one or more tokens representing the action to be performed by the agent at the planning time step.

### Claim 14 (depends on 1)

The method of claim 1, wherein the sequence-processing neural network has been trained on a plurality of training sequences of tokens from the vocabulary to optimize a task objective that comprises a next token prediction objective.

### Claim 15 (depends on 14)

The method of claim 14, wherein one or more of the training sequences of tokens are transition sequences representing interactions with the environment.

### Claim 16 (depends on 14)

The method of claim 14, wherein one or more of the training sequences of tokens are transition sequences representing interactions of a second agent with a second environment, wherein the second agent is different from the agent, the second environment is different from the environment, or both.

### Claim 17 (depends on 15)

The method of claim 15, wherein the auto-regressive neural network has been pre-trained on training sequences representing interactions of one or more agents with one or more different environments and fine-tuned on the transition sequences representing interactions of the agent with the environment.

### Claim 18 (depends on 9)

The method of claim 9, wherein the first input sequence further comprises one or more separator tokens from the vocabulary that separate tokens corresponding to different planning time steps.

### Claim 19 (depends on 1)

The method of claim 1, wherein processing the first input sequence for the planning time step using the sequence-processing neural network to generate the output sequence that comprises a sequence of tokens from the vocabulary of tokens that represents the observation at the subsequent planning time step in the planned trajectory comprises, for each position starting from a first position in the output sequence:
processing a current input sequence using the sequence-processing neural network to generate a score distribution over the vocabulary of tokens, wherein for the first position the current input sequence comprises the first input sequence and wherein for each subsequent position the current input sequence comprises the first input sequence followed by the tokens at each preceding position in the output sequence;
selecting a token from the vocabulary of tokens in accordance with the score distribution; and
updating the current input sequence by appending the selected token to the current input sequence.

### Claim 20 (depends on 1)

The method of claim 1, wherein the sequence-processing neural network is a self-attention neural network that comprises one or more causally masked self-attention layers.

### Claim at position 21

21-29. (canceled)

### Claim 30

A system comprising one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the one more computers to perform operations for controlling an agent interacting with an environment to perform a task episode, the operations comprising:
at each of a plurality of current time steps in the task episode:
obtaining an observation characterizing a current state of the environment at the time step;
generating a plurality of planned trajectories starting from the current state of the environment, wherein each planned trajectory comprises data representing a respective observation at each of a sequence of planning time steps that starts at the current time step and includes a plurality of future time steps that are each after the current time step, the generating comprising, for each planned trajectory and at each planning time step other than a last planning time step in the sequence:
identifying an action to be performed by the agent in response to the observation at the planning time step in the planned trajectory; and
generating data specifying the observation at the subsequent planning time step in the planned trajectory, comprising processing a first input sequence for the planning time step that comprises a sequence of tokens from a vocabulary of tokens and representing the respective observation in the planned trajectory at the planning time step and one or more tokens from the vocabulary of tokens and representing the identified action at the planning time step, using a sequence-processing neural network, to generate an output sequence that comprises a sequence of tokens from the vocabulary of tokens that represents the observation at the subsequent planning time step in the planned trajectory;
selecting a planned trajectory from the plurality of planned trajectories; and
causing the agent to perform the identified action for the first planning time step in the selected planned trajectory.

### Claim 31

One or more non-transitory computer storage media storing instructions that when executed by one or more computers cause the one more computers to perform operations for controlling an agent interacting with an environment to perform a task episode, the operations comprising:
at each of a plurality of current time steps in the task episode:
obtaining an observation characterizing a current state of the environment at the time step;
generating a plurality of planned trajectories starting from the current state of the environment, wherein each planned trajectory comprises data representing a respective observation at each of a sequence of planning time steps that starts at the current time step and includes a plurality of future time steps that are each after the current time step, the generating comprising, for each planned trajectory and at each planning time step other than a last planning time step in the sequence:
identifying an action to be performed by the agent in response to the observation at the planning time step in the planned trajectory; and
generating data specifying the observation at the subsequent planning time step in the planned trajectory, comprising processing a first input sequence for the planning time step that comprises a sequence of tokens from a vocabulary of tokens and representing the respective observation in the planned trajectory at the planning time step and one or more tokens from the vocabulary of tokens and representing the identified action at the planning time step, using a sequence-processing neural network, to generate an output sequence that comprises a sequence of tokens from the vocabulary of tokens that represents the observation at the subsequent planning time step in the planned trajectory;
selecting a planned trajectory from the plurality of planned trajectories; and
causing the agent to perform the identified action for the first planning time step in the selected planned trajectory.
