# US 2025/0165532 A1 · Fast exploration and learning of latent graph models

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2025/0165532 A1
- Document ID: US20250165532A1
- Status: Published application
- Kind: A1
- Application number: 18/959,423
- Filed: 2024-11-25
- Published: 2025-05-22
- Earliest claimed: 2022-09-28
- Assignees: DeepMind Technologies Limited
- Applicants: DeepMind Technologies Limited
- Inventors: Sivaramakrishnan Swaminathan; Meet Kirankumar Dave; Miguel Lazaro-Gredilla; Dileep George
- Web page: https://patentgrep.com/patent/US20250165532A1

## CPC

- G06F 16/9024: {Graphs; Linked lists (G06F16/9027 takes precedence)}
- G06F 17/12: Simultaneous equations {, e.g. systems of linear equations}
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for generating a graph model representing an environment being interacted with by an agent. In one aspect, one of the methods include: obtaining experience data; using the experience data to update a visitation count for each of one or more state-action pairs represented by the graph model; and at each of multiple environment exploration steps: computing a utility measure for each of the one or more state-action pairs represented by the graph model; determining, based on the utility measures, a sequence of one or more planned actions that have an information gain that satisfies a threshold; and controlling the agent to perform the sequence of one or more planned actions to cause the environment to transition from a state characterized by a last observation received after a last action in the experience data into a different state.

## Claims

### Claim 1

(canceled)

### Claim 2

A method performed by one or more computers, wherein the method comprises:
obtaining experience data generated as a result of controlling an agent in an environment to perform a sequence of one or more actions from a possible set of actions, each action being performed in response to receiving a respective observation characterizing a respective state of the environment;
using the experience data to update a visitation count for each of one or more state-action pairs, wherein each state-action pair represents a respective state of the environment and a respective action performed by the agent; and
performing an environment exploration step to explore the environment, the performing comprising:
computing a utility measure for each of the one or more state-action pairs, wherein computing the utility measure comprises evaluating a closed form utility function using at least the updated visitation counts;
determining, based on the utility measures, a sequence of one or more planned actions that have an information gain that satisfies a threshold; and
controlling the agent to perform the sequence of one or more planned actions to cause the environment to transition from a state characterized by a last observation received after a last action in the experience data into a different state.

### Claim 3 (depends on 2)

The method of claim 2, wherein determining the sequence of one or more planned actions comprises:
selecting, from all state-action pairs that includes a node representing the state characterized by the last observation, a selected state-action pair that has a highest utility measure; and
using an action included in the selected state-action pair as one single planned action.

### Claim 4 (depends on 2)

The method of claim 2, wherein determining the sequence of one or more planned actions comprises:
using a reinforcement learning technique to generate an action selection policy for the node representing the state characterized by the last observation; and
using the action selection policy to select a sequence of multiple planned actions.

### Claim 5 (depends on 4)

The method of claim 4, wherein the reinforcement learning technique comprises a value iteration technique that maximizes a total utility measure of actions selected by the action selection policy.

### Claim 6 (depends on 2)

The method of claim 2, further comprising, at each of multiple environment exploration steps:
obtaining updated experience data generated as a result of controlling the agent to perform the sequence of one or more planned actions; and
using the updated experience data to update the visitation count for each of one or more state-action pairs.

### Claim 7 (depends on 2)

The method of claim 2, wherein the information gain of the sequence of one or more planned actions comprises a combination of utility measures of the planned actions included in the sequence.

### Claim 8 (depends on 2)

The method of claim 2, wherein the sequence of one or more actions comprise one or more randomly selected actions from the possible set of actions.

### Claim 9 (depends on 2)

The method of claim 2, wherein the agent comprises a mechanical agent and the environment is a real-world environment.

### Claim 10 (depends on 2)

The method of claim 2, wherein the agent comprises a computer program and the environment is a simulated environment.

### Claim 11

A method performed by one or more computers for exploring an environment being interacted with by an agent, wherein the method comprises:
computing a utility measure for each of one or more state-action pairs in a representation of the environment based on evaluating a closed form utility function, wherein each state-action pair represents a respective state of the environment and a respective action performed by the agent;
determining, based on the utility measures, a sequence of one or more planned actions that have an information gain that satisfies a threshold;
controlling the agent to perform the sequence of one or more planned actions to cause the environment to transition into one or more different states;
obtaining updated experience data generated as a result of controlling the agent to perform the sequence of one or more planned actions; and
using the updated experience data to update the representation of the environment.

### Claim 12 (depends on 11)

The method of claim 11, wherein determining the sequence of one or more planned actions comprises:
selecting, from all state-action pairs that includes a node representing the state characterized by the last observation, a selected state-action pair that has a highest utility measure; and
using an action included in the selected state-action pair as one single planned action.

### Claim 13 (depends on 11)

The method of claim 11, wherein determining the sequence of one or more planned actions comprises:
using a reinforcement learning technique to generate an action selection policy for the node representing the state characterized by the last observation; and
using the action selection policy to select a sequence of multiple planned actions.

### Claim 14 (depends on 13)

The method of claim 13, wherein the reinforcement learning technique comprises a value iteration technique that maximizes a total utility measure of actions selected by the action selection policy.

### Claim 15 (depends on 11)

The method of claim 11, further comprising, at each of multiple environment exploration steps:
obtaining updated experience data generated as a result of controlling the agent to perform the sequence of one or more planned actions; and
using the updated experience data to update the visitation count for each of one or more state-action pairs.

### Claim 16 (depends on 11)

The method of claim 11, wherein the information gain of the sequence of one or more planned actions comprises a combination of utility measures of the planned actions included in the sequence.

### Claim 17 (depends on 11)

The method of claim 11, wherein the sequence of one or more actions comprise one or more randomly selected actions from the possible set of actions.

### Claim 18

A method performed by one or more computers, wherein the method comprises:
performing one or more environment exploration steps to update a representation of an environment being interacted with by an agent, comprising, at each of the one or more environment exploration steps:
computing a utility measure for each of one or more state-action pairs in the representation of an environment based on evaluating a closed form utility function, wherein each state-action pair represents a respective state of the environment and a respective action performed by the agent;
determining, based on the utility measures, a sequence of one or more planned actions that have an information gain that satisfies a threshold;
controlling the agent to perform the sequence of one or more planned actions to cause the environment to transition into one or more different states;
obtaining updated experience data generated as a result of controlling the agent to perform the sequence of one or more planned actions; and
using the updated experience data to update the representation of the environment; and
controlling the agent to perform a task by causing the agent to perform actions selected from a possible set of actions by using the updated representation of the environment.

### Claim 19 (depends on 18)

The method of claim 18, wherein the agent comprises a mechanical agent and the environment is a real-world environment.

### Claim 20 (depends on 18)

The method of claim 18, wherein the agent comprises a computer program and the environment is a simulated environment.

### Claim 21 (depends on 18)

The method of claim 18, wherein determining the sequence of one or more planned actions comprises:
selecting, from all state-action pairs that includes a node representing the state characterized by the last observation, a selected state-action pair that has a highest utility measure; and
using an action included in the selected state-action pair as one single planned action.
