# US 2021/0063974 A1 · Method for reinforcement learning, recording medium storing reinforcement learning program, and reinforcement learning apparatus

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0063974 A1
- Document ID: US20210063974A1
- Status: Published application
- Kind: A1
- Application number: 17/001,706
- Filed: 2020-08-25
- Published: 2021-03-04
- Earliest claimed: 2019-08-27
- Assignees: Fujitsu Limited
- Applicants: Fujitsu Limited
- Inventors: Yoshihiro OKAWA; Tomotake Sasaki; Hidenao Iwane; Hitoshi Yanami
- Web page: https://patentgrep.com/patent/US20210063974A1

## CPC

- G06N 3/092: Reinforcement learning
- G06N 3/006: based on simulated virtual individual or collective life forms, e.g. social simulations or particle swarm optimisation [PSO]
- G06N 5/01: Dynamic search techniques; Heuristics; Dynamic trees; Branch-and-bound
- G06N 5/02: Knowledge representation; Symbolic representation
- G06N 7/01: Probabilistic graphical models, e.g. probabilistic networks
- G05B 13/042: {in which a parameter or coefficient is automatically adjusted to optimise the performance}
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G05B: CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS (systems for controlling or regulating non-electric variables G05D; systems for regulating electric or magnetic variables G05F; control devices or systems insofar as characterised by mechanical features only G05G)

## Abstract

A method for reinforcement learning performed by a computer is disclosed. The method includes: predicting a state of a target to be controlled in reinforcement learning at each time point to measure a state of the target, the time point being included in a period from a time point to determine a present action to a time point to determine a subsequent action; calculating a degree of risk concerning the state of the target at the each time point with respect to a constraint condition based on a result of prediction; specifying a search range concerning the present action to the target in accordance with the calculated degree of risk and a degree of impact of the present action to the target on the state of the target at the each time point; and determining the present action to the target based on the specified search range.

## Claims

### Claim 1

A method for reinforcement learning of causing a computer to execute a process comprising:
predicting a state of a target to be controlled in reinforcement learning at each time point to measure a state of the target, the time point being included in a period from after a time point to determine a present action to a time point not later than determination of a subsequent action, on a condition that a time interval to measure the state of the target is different from a time interval to determine the action to the target;
calculating a degree of risk concerning the state of the target at the each time point with respect to a constraint condition concerning the state of the target based on a result of prediction of the state of the target;
specifying a search range concerning the present action to the target in accordance with the calculated degree of risk concerning the state of the target at the each time point and a degree of impact of the present action to the target on the state of the target at the each time point; and
determining the present action to the target based on the specified search range concerning the present action to the target.

### Claim 2 (depends on 1)

The method according to claim 1, wherein
in the specifying, the search range is specified such a way as to become narrower as the degree of risk is higher and to become narrower as the degree of impact is higher.

### Claim 3 (depends on 1)

The method according to claim 1, wherein the time interval to determine the action to the target is longer than the time interval to measure the state of the target.

### Claim 4 (depends on 1)

The method according to claim 1, wherein
in the specifying, the search range is specified under a probabilistic evaluation index concerning satisfaction of the constraint condition, and
in the determining, the present action to the target is stochastically determined based on the specified search range.

### Claim 5 (depends on 1)

The method according to claim 1, wherein
in the specifying, the search range is specified under a probabilistic evaluation index concerning satisfaction of the constraint condition when the calculated degree of risk concerning the state of the target at each time point is below a threshold, and
in the determining, a prescribed value is determined for the action to the target when the calculated degree of risk concerning the state of the target at a certain time point included in the period is equal to or above a threshold and the present action to the target is stochastically determined based on the specified search range when the calculated degree of risk concerning the state of the target at each time point is below the threshold.

### Claim 6 (depends on 5)

The method according to claim 5, wherein
in the specifying, when the calculated degree of risk concerning the state of the target at each time point is below the threshold, an mean value applicable to the present action to the target is calculated, then a variance-covariance matrix is calculated under a probabilistic evaluation index concerning satisfaction of the constraint condition in accordance with the calculated degree of risk concerning the state of the target at each time point and a degree of impact of the present action to the target on the state of the target at each time point, and the search range is specified by using the calculated mean value and the calculated variance-covariance matrix.

### Claim 7 (depends on 5)

The method according to claim 5, wherein the prescribed value is equal to 0.

### Claim 8 (depends on 1)

The method according to claim 1, wherein the constraint condition is linear relative to the state of the target.

### Claim 9 (depends on 1)

The method according to claim 1, wherein the state of the target at each time point included in the period is predicted by using previous knowledge concerning the target.

### Claim 10 (depends on 9)

The method according to claim 9, wherein
the target is a power generation facility,
the previous knowledge is information based on at least any of
a specification value of the power generation facility,
a nominal value of a parameter applied to the power generation facility, and
an allowance of the parameter applied to the power generation facility,
each of the calculating, the specifying, and the determining is executed in course of reinforcement learning to learn a policy to control the target by
defining power generator torque in the power generation facility as the action,
defining at least any of an amount of power generation in the power generation facility, an amount of revolutions of a turbine in the power generation facility, a revolving speed of the turbine in the power generation facility, a direction of wind at the power generation facility, and a wind velocity at the power generation facility as the state, and
defining the amount of power generation in the power generation facility as a reward,
the time interval to measure the state of the target is a time interval to measure at least any of the amount of power generation in the power generation facility, the amount of revolutions of the turbine in the power generation facility, the revolving speed of the turbine in the power generation facility, the direction of wind at the power generation facility, and the wind velocity at the power generation facility, and
the time interval to determine the action to the target is a time interval to determine the power generator torque in the power generation facility.

### Claim 11 (depends on 9)

The method according to claim 9, wherein
the target is an air-conditioning facility,
the previous knowledge is information based on at least any of
a specification value of the air-conditioning facility,
a nominal value of a parameter applied to the air-conditioning facility, and
an allowance of the parameter applied to the air-conditioning facility,
each of the calculating, the specifying, and the determining is executed in course of reinforcement learning to learn a policy to control the target by
defining at least any of a set temperature of the air-conditioning facility and a set air volume of the air-conditioning facility as the action,
defining at least any of a temperature inside a room where the air-conditioning facility is installed, a temperature outside the room where the air-conditioning facility is installed, and a weather as the state, and
defining an amount of power consumption by the air-conditioning facility as a cost,
the time interval to measure the state of the target is a time interval to measure at least any of the temperature inside the room where the air-conditioning facility is installed, the temperature outside the room where the air-conditioning facility is installed, and the weather, and
the time interval to determine the action to the target is a time interval to determine at least any of the set temperature of the air-conditioning facility and the set air volume of the air-conditioning facility.

### Claim 12 (depends on 9)

The method according to claim 9, wherein
the target is an industrial robot,
the previous knowledge is information based on at least any of
a specification value of the industrial robot,
a nominal value of a parameter applied to the industrial robot, and
an allowance of the parameter applied to the industrial robot,
each of the calculating, the specifying, and the determining is executed in course of reinforcement learning to learn a policy to control the target by
defining motor torque of the industrial robot as the action,
defining at least any of a shot image of the industrial robot, a position of a joint of the industrial robot, an angle of the joint of the industrial robot, and an angular velocity of the joint of the industrial robot as the state, and
defining an amount of production of products by the industrial robot as a reward,
the time interval to measure the state of the target is a time interval to measure at least any of the shot image of the industrial robot, the position of the joint of the industrial robot, the angle of the joint of the industrial robot, and the angular velocity of the joint of the industrial robot, and
the time interval to determine the action to the target is a time interval to determine the motor torque of the industrial robot.

### Claim 13 (depends on 9)

The method according to claim 9, wherein
the previous knowledge includes model information expressed by subjecting a function of the state of the target at each time point to measure the state of the target, the time point being included in the period from after the time point to determine the present action to the time point not later than determination of the subsequent action, to linear approximation while using a variable indicating the state of the target and a variable indicating the action to the target at the time point to determine the present action.

### Claim 14 (depends on 13)

The method according to claim 13, wherein
in the predicting, a predicted value of the state of the target is calculated based on the model information and on an upper limit of an error included in the predicted value at each time point to measure the state of the target, the time point being included in the period from after the time point to determine the present action to the time point not later than determination of the subsequent action.

### Claim 15 (depends on 1)

The method according to claim 1, wherein
each of the calculating, the specifying, and the determining is executed in course of reinforcement learning of an episode type in terms of any of a period from a point of initialization of the state of the target to a point of discontinuation of satisfaction of the constraint condition by the state of the target and a period from the point of initialization of the state of the target to a lapse of a given length of time.

### Claim 16 (depends on 1)

The method according to claim 1, wherein the target has a property that the state of the target is guaranteed to satisfy the constraint condition at the time point when subsequent measurement of the state takes place on a condition that the state of the target satisfies the constraint condition and the action to the target has a value of 0 at a certain time point to measure the state.

### Claim 17

A non-transitory computer-readable storage medium having stored a reinforcement learning program for causing a computer to execute a process comprising:
predicting a state of a target to be controlled in reinforcement learning at each time point to measure a state of the target, the time point being included in a period from after a time point to determine a present action to a time point not later than determination of a subsequent action, on a condition that a time interval to measure the state of the target is different from a time interval to determine the action to the target;
calculating a degree of risk concerning the state of the target at the each time point with respect to a constraint condition concerning the state of the target based on a result of prediction of the state of the target;
specifying a search range concerning the present action to the target in accordance with the calculated degree of risk concerning the state of the target at the each time point and a degree of impact of the present action to the target on the state of the target at the each time point; and
determining the present action to the target based on the specified search range concerning the present action to the target.

### Claim 18

A reinforcement learning apparatus comprising:
a memory, and
a processor coupled to the memory and configured to:
predict a state of a target to be controlled in reinforcement learning at each time point to measure a state of the target, the time point being included in a period from after a time point to determine a present action to a time point not later than determination of a subsequent action, on a condition that a time interval to measure the state of the target is different from a time interval to determine the action to the target;
calculate a degree of risk concerning the state of the target at the each time point with respect to a constraint condition concerning the state of the target based on a result of prediction of the state of the target;
specify a search range concerning the present action to the target in accordance with the calculated degree of risk concerning the state of the target at the each time point and a degree of impact of the present action to the target on the state of the target at the each time point; and
determine the present action to the target based on the specified search range concerning the present action to the target.
