# US 2024/0202570 A1 · Enabling classical planning in interactive environments with incomplete models

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2024/0202570 A1
- Document ID: US20240202570A1
- Status: Published application
- Kind: A1
- Application number: 18/066,270
- Filed: 2022-12-14
- Published: 2024-06-20
- Earliest claimed: 2022-12-14
- Assignees: International Business Machines Corporation
- Applicants: International Business Machines Corporation
- Inventors: Don Joven Ravoy Agravante; Michiaki Tatsubori
- Web page: https://patentgrep.com/patent/US20240202570A1

## CPC

- G06N 20/00: Machine learning
- G06N 3/006: based on simulated virtual individual or collective life forms, e.g. social simulations or particle swarm optimisation [PSO]
- G06N 7/01: Probabilistic graphical models, e.g. probabilistic networks
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

A computer-implemented method in a model-based reinforcement learning (RL) system with logic states includes switching an agent between a first mode and a second mode, the first mode being a probabilistic planning mode and the second mode being an information gathering mode. In response to the agent being in the probabilistic planning mode, the agent computes a predictive state representation, given a history of observations and actions taken, and the agent scores action candidates based on planning with the predictive state representation so that actions with resolved plans with confidence to achieve a goal state are scored higher than actions without resolved plans. In response to the agent being in the information gathering mode, the agent scores action candidates based on a Q function of a value of expected information to be gathered from a given pair of state and action.

## Claims

### Claim 1

A computer-implemented method in a model-based reinforcement learning (RL) system with logic states, the method comprising:
switching an agent between a first mode and a second mode, the first mode being a probabilistic planning mode and the second mode being an information gathering mode;
in response to the agent being in the probabilistic planning mode, the agent:
computes a predictive state representation, given a history of observations and actions taken; and
scores action candidates based on planning with the predictive state representation such that actions with resolved plans with confidence to achieve a goal state are scored higher than actions without resolved plans; and
in response to the agent being in the information gathering mode, the agent scores action candidates based on a Q function of a value of expected information to be gathered from a given pair of state and action.

### Claim 2 (depends on 1)

The computer-implemented method according to claim 1, wherein:
the Q function is learned from trajectories in an environment; and
an increase in a belief state certainty is given as an intrinsic reward.

### Claim 3 (depends on 2)

The computer-implemented method according to claim 2, wherein the intrinsic reward is an information gain reward based on a local measure of information gain between a current state and a next state of the environment.

### Claim 4 (depends on 3)

The computer-implemented method of claim 3, wherein the information gain reward uses a number of novel facts in the next state compared to the current state.

### Claim 5 (depends on 1)

The computer-implemented method according to claim 1, wherein, in response to a situation where a path cannot be found in a classical planning problem, the agent switches from the probabilistic planning mode to the information gathering mode.

### Claim 6 (depends on 1)

The computer-implemented method according to claim 1, wherein:
the agent is provided with a goal state; and
the agent is in an environment with partial observability without a path to reach the goal state.

### Claim 7 (depends on 1)

The computer-implemented method according to claim 1, wherein:
the agent is provided with a goal state; and
the agent is in an incomplete model that precludes a path to reach the goal state.

### Claim 8 (depends on 1)

The computer-implemented method according to claim 1, further comprising learning or updating, with a learning algorithm, action models within the probabilistic planning mode using one or more previous interactions with the environment.

### Claim 9 (depends on 1)

The computer-implemented method according to claim 1, wherein an initial action model of an environment is given to the agent by one or more domain experts.

### Claim 10

A method for planning in interactive environments with partial observability and incomplete models, the method comprising:
producing, with a classical planning algorithm, a path from a given initial state to a goal state in response to a model of a world being known to an agent;
taking action to increase knowledge about a current state of the world with an information gathering policy trained to take actions that increase knowledge about the current state of the world; and
switching, with an online replanning framework, between using the classical planning algorithm and using the information gathering policy based on a current model of the world.

### Claim 11 (depends on 10)

The method of claim 10, further comprising learning or updating, with a learning algorithm, action models within the classical planning algorithm using one or more previous interactions with the environment.

### Claim 12 (depends on 10)

The method according to claim 10, wherein an initial action model of an environment is given to the agent by one or more domain experts.

### Claim 13 (depends on 10)

The method of claim 10, further comprising learning, with a Q-learning algorithm, from rewards obtained during an interaction, a value function that scores state-action pairs.

### Claim 14 (depends on 10)

The method of claim 10, wherein:
the Q-learning algorithm is learned from trajectories in the environment; and
an increase in a belief state certainty is given as an intrinsic reward.

### Claim 15 (depends on 14)

The method according to claim 14, wherein the intrinsic reward is an information gain reward defined based on a local measure of information gain between a current state and a next state of the environment.

### Claim 16 (depends on 15)

The method of claim 15, wherein the information gain reward uses a number of novel facts in the next state compared to a number of facts known in the current state.

### Claim 17

A non-transitory computer readable storage medium tangibly embodying a computer readable program code having computer readable instructions that, when executed, causes a computer device to carry out a method for planning in interactive environments with partial observability and incomplete models, the method comprising:
producing, with a classical planning algorithm, a path from a given initial state to a goal state in response to a model of a world being known to an agent;
taking action to increase knowledge about a current state of the world with an information gathering policy trained to take actions that increase knowledge about the current state of the world; and
switching, with an online replanning framework, between using the classical planning algorithm and using the information gathering policy based on a current model of the world.

### Claim 18 (depends on 17)

The non-transitory computer readable storage medium of claim 17, the method further comprising learning or updating, with a learning algorithm, action models within the classical planning algorithm using one or more previous interactions with the environment.

### Claim 19 (depends on 17)

The non-transitory computer readable storage medium of claim 17, the method further comprising giving an initial action model of an environment to the agent by one or more domain experts.

### Claim 20 (depends on 17)

The non-transitory computer readable storage medium of claim 17, the method further comprising learning, with a Q-learning algorithm, from rewards obtained during an interaction, a value function that scores state-action pairs, wherein the Q-learning algorithm is learned from trajectories in the environment, and an increase in a belief state certainty is given as an intrinsic reward.
