# US 2024/0202537 A1 · Learning method, learning device, control method, control device, and storage medium

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2024/0202537 A1
- Document ID: US20240202537A1
- Status: Published application
- Kind: A1
- Application number: 18/460,113
- Filed: 2023-09-01
- Published: 2024-06-20
- Earliest claimed: 2022-12-19
- Assignees: Kabushiki Kaisha Toshiba
- Applicants: Kabushiki Kaisha Toshiba
- Inventors: Ryosuke NONAKA; Toshimitsu KANEKO
- Web page: https://patentgrep.com/patent/US20240202537A1

## CPC

- G06N 3/092: Reinforcement learning
- G06N 3/006: based on simulated virtual individual or collective life forms, e.g. social simulations or particle swarm optimisation [PSO]
- G06N 3/08: Learning methods
- G06N 7/01: Probabilistic graphical models, e.g. probabilistic networks
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

According to one embodiment, a learning method includes calculating a probability distribution indicating a distribution of a probability density or a distribution of a probability at which actions are selected, selecting a first action based on the probability distribution, causing a control target to execute the first action, receiving a reward and next observation data, calculating a probability density or a probability of the first action, correcting the reward, and updating the control parameter. The reward is corrected such that the reward increases as the probability density or probability decreases.

## Claims

### Claim 1

A learning method comprising:
a first step of receiving current observation data;
a second step of calculating a probability distribution indicating a distribution of a probability density or a distribution of a probability at which actions are selected, based on the current observation data and a control parameter;
a third step of selecting a first action among the actions based on the probability distribution;
a fourth step of causing a control target to execute the first action;
a fifth step of receiving a first reward and next observation data observed after the control target has executed the first action;
a sixth step of calculating a probability density or a probability of the first action from the probability distribution;
a seventh step of correcting the first reward based on a probability density of the first action or a probability of the first action; and
an eighth step of updating the control parameter based on the current observation data, the first action, the next observation data, and the corrected first reward,
wherein the seventh step comprises correcting the first reward such that the first reward increases as the probability density or probability decreases.

### Claim 2 (depends on 1)

The learning method of claim 1, wherein the eighth step comprises updating the control parameter for each control period of the control target.

### Claim 3 (depends on 1)

The learning method of claim 1, wherein the second step comprises inputting the current observation data to a neural network whose input/output characteristics vary according to the control parameter, the neural network outputting the probability distribution.

### Claim 4 (depends on 1)

The learning method of claim 1, wherein the first reward indicates whether selection of the first action is appropriate.

### Claim 5 (depends on 1)

The learning method of claim 1, wherein:
the seventh step comprises correcting the first reward by adding a second reward to the first reward; and
the second reward increases as the probability density or probability decreases.

### Claim 6 (depends on 1)

The learning method of claim 1, wherein:
the seventh step comprises correcting the first award by multiplying the first reward by a factor; and
the factor increases as the probability density or probability decreases.

### Claim 7

A learning device comprising a processor configured to:
receive current observation data;
calculate a probability distribution indicating a distribution of a probability density or a distribution of a probability at which actions are selected, based on the current observation data and a control parameter;
select a first action among the actions based on the probability distribution;
cause a control target to execute the first action;
receive a first reward and next observation data observed after the control target has executed the first action;
calculate a probability density or a probability of the first action from the probability distribution;
correct the first reward based on a probability density of the first action or a probability of the first action; and
correct the control parameter based on the current observation data, the first action, the next observation data, and the corrected first reward,
wherein processor is configured to correct the first reward such that the first reward increases as the probability density or probability decreases.

### Claim 8

A non-transitory computer-readable storage medium storing computer-executable instructions that, when executed, cause the computer to:
receive current observation data;
calculate a probability distribution indicating a distribution of a probability density or a distribution of a probability at which actions are selected, based on the current observation data and a control parameter;
select a first action among the actions based on the probability distribution;
cause a control target to execute the first action;
receive a first reward and next observation data observed after the control target has executed the first action;
calculate a probability density or a probability of the first action from the probability distribution;
correct the first reward based on a probability density of the first action or a probability of the first action; and
correct the control parameter based on the current observation data, the first action, the next observation data, and the corrected first reward,
wherein processor is configured to correct the first reward such that the first reward increases as the probability density or probability decreases.

### Claim 9 (depends on 1)

A control method comprising:
a first step of receiving current observation data;
a second step of calculating a probability distribution indicating a distribution of a probability density or a distribution of a probability at which actions are selected, based on the current observation data and a control parameter updated by the learning method of claim 1;
a third step of selecting a first action among the actions based on the probability distribution; and
a fourth step of causing a control target to execute the first action.

### Claim 10

A control device comprising a processor configured to:
receive current observation data;
calculate a probability distribution indicating a distribution of a probability density or a distribution of a probability at which actions are selected, based on the current observation data and a control parameter;
select a first action among the actions based on the probability distribution; and
cause a control target to execute the first action, wherein
the control parameter is calculated based on the current observation data and the control parameter,
the control parameter is updated based on the probability distribution and a reward indicating whether selection of the first action is appropriate, and
the reward is corrected such that the reward increases as the probability density or probability decreases.

### Claim 11

A non-transitory computer-readable storage medium storing computer-executable instructions that, when executed, cause a computer to:
receive current observation data;
calculate a probability distribution indicating a distribution of a probability density or a distribution of a probability at which actions are selected, based on the current observation data and a control parameter;
select a first action among the actions based on the probability distribution; and
cause a control target to execute the first action, wherein
the control parameter is calculated based on the current observation data and the control parameter,
the control parameter is updated based on the probability distribution and a reward indicating whether selection of the first action is appropriate, and
the reward is corrected such that the reward increases as the probability density or probability decreases.
