# US 2025/0384342 A1 · Learning device, learning method, and recording medium

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2025/0384342 A1
- Document ID: US20250384342A1
- Status: Published application
- Kind: A1
- Application number: 19/233,160
- Filed: 2025-06-10
- Published: 2025-12-18
- Earliest claimed: 2024-06-17
- Assignees: NEC Corporation
- Applicants: NEC Corporation
- Inventors: Yuki NAKAGUCHI
- Web page: https://patentgrep.com/patent/US20250384342A1

## CPC

- G06N 20/00: Machine learning
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

In a learning device, a generation means generates a value estimation model from preference data indicating combinations of each state and action. An acquisition means acquires a next state as an execution result of an action determined using a strategy of a learning target model. An estimation means estimates a state value or action value of a next state using the next state and the value estimation model. A strategy update means updates the strategy of the learning target model using the state value or the action value. Accordingly, it is possible to realize interactive imitation learning that can be performed with offline data indicating preferences for a teacher model.

## Claims

### Claim 1

A learning device comprising:
at least one memory configured to store instructions; and
at least one processor configured to execute the instructions to:
generate a value estimation model from preference data indicating combinations of each state and action;
acquire a next state as an execution result of an action determined using a strategy of a learning target model; and
estimate a state value or action value of a next state using the next state and the value estimation model; and
update the strategy of the learning target model using the state value or the action value.

### Claim 2 (depends on 1)

The learning device according to claim 1, wherein the at least one processor
generates a probability model indicating a probability of preference between the combinations of each state and action,
parametrizes the value estimation model and the probability model with a common parameter, and
generates the value estimation model by optimizing the common parameter using the preference data.

### Claim 3 (depends on 2)

The learning device according to claim 2, wherein the at least one processor
considers the probability model as a probability model of binary classification, which selects one combination from two combinations of each state and action at a given time, and
optimizes the common parameter to minimize a loss of the binary classification.

### Claim 4 (depends on 1)

The learning device according to claim 1, wherein the at least one processor updates the strategy of the learning target model by interactive imitation learning.

### Claim 5 (depends on 4)

The learning device according to claim 4, wherein the at least one processor estimates the action value of a next state using the next state and the value estimation mode, in a case where the interactive imitation learning uses the action value.

### Claim 6 (depends on 4)

The learning device according to claim 4, wherein the at least one processor estimates the state value of a next state in a case where the interactive imitation learning uses the state value.

### Claim 7 (depends on 6)

The learning device according to claim 6, wherein the at least one processor estimates the state value of the next state using the next state, the value estimation model, a first expression representing a relationship between the value estimation model and the state value.

### Claim 8 (depends on 6)

The learning device according to claim 6, wherein the at least one processor
generates a model of the strategy from the preference data, and
estimates the state value of the next state using the next state, the value estimation model, the model of the strategy, a second expression representing a relationship between the value estimation model, the strategy, and the state value.

### Claim 9

A learning method performed by a computer, comprising:
generating a value estimation model from preference data indicating combinations of each state and action;
acquiring a next state as an execution result of an action determined using a strategy of a learning target model;
estimating a state value or action value of a next state using the next state and the value estimation model; and
updating the strategy of the learning target model using the state value or the action value.

### Claim 10

A non-transitory computer-readable recording medium storing a program causing a computer to execute processing of:
generating a value estimation model from preference data indicating combinations of each state and action;
acquiring a next state as an execution result of an action determined using a strategy of a learning target model;
estimating a state value or action value of a next state using the next state and the value estimation model; and
updating the strategy of the learning target model using the state value or the action value.
