# US 2012/0253514 A1 · Reinforcement learning apparatus, control apparatus, and reinforcement learning method

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2012/0253514 A1
- Document ID: US20120253514A1
- Status: Published application
- Kind: A1
- Application number: 13/432,094
- Filed: 2012-03-28
- Published: 2012-10-04
- Earliest claimed: 2011-03-30
- Assignees: Advanced Telecommunications Research Institute International; Honda Motor Co., Ltd.
- Inventors: Norikazu Sugimoto; Yugo Ueda; Tadaaki Hasegawa; Soshi Iba; Koji Akatsuka
- Web page: https://patentgrep.com/patent/US20120253514A1

## CPC

- G05B 13/0265: {the criterion being a learning criterion}
- Y10S 901/03: Teaching system
- G05B: CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS (systems for controlling or regulating non-electric variables G05D; systems for regulating electric or magnetic variables G05F; control devices or systems insofar as characterised by mechanical features only G05G)
- Y10S: TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS

## Abstract

It is possible to perform robot motor learning in a quick and stable manner using a reinforcement learning apparatus including: a first-type environment parameter obtaining unit that obtains a value of one or more first-type environment parameters; a control parameter value calculation unit that calculates a value of one or more control parameters maximizing a reward by using the value of the one or more first-type environment parameters; a control parameter value output unit that outputs the value of the one or more control parameters to the control object; a second-type environment parameter obtaining unit that obtains a value of one or more second-type environment parameters; a virtual external force calculation unit that calculates the virtual external force by using the value of the one or more second-type environment parameters; and a virtual external force output unit that outputs the virtual external force to the control object.

## Claims

### Claim 1

A reinforcement learning apparatus comprising:
a reward function storage unit in which a reward function for outputting a reward can be stored;
a first-type environment parameter obtaining unit that obtains a value of one or more first-type environment parameters relating to an environment of a moving control object;
a control parameter value calculation unit that calculates a value of one or more control parameters maximizing a reward that is output by the reward function by substituting the value of the one or more first-type environment parameters into the reward function;
a control parameter value output unit that outputs the value of the one or more control parameters to the control object;
a virtual external force function storage unit in which a virtual external force function for outputting a virtual external force can be stored,
a second-type environment parameter obtaining unit that obtains a value of one or more second-type environment parameters relating to the virtual external force;
a virtual external force calculation unit that calculates the virtual external force by substituting the value of the one or more second-type environment parameters into the virtual external force function; and
a virtual external force output unit that outputs the virtual external force to the control object.

### Claim 2 (depends on 1)

The reinforcement learning apparatus according to claim 1, comprising a reinforcement learner and a virtual external force generator that is separated from the reinforcement learner,
wherein the reinforcement learner comprises:
the reward function storage unit, the first-type environment parameter obtaining unit, the control parameter value calculation unit, and the control parameter value output unit, and
the virtual external force generator comprises:
the virtual external force function storage unit, the second-type environment parameter obtaining unit, the virtual external force calculation unit, and the virtual external force output unit.

### Claim 3 (depends on 1)

The reinforcement learning apparatus according to claim 1, wherein the value of the one or more control parameters that is output by the control parameter value output unit and the virtual external force that is output by the virtual external force output unit are added together, and given to the control object.

### Claim 4 (depends on 2)

The reinforcement learning apparatus according to claim 2, wherein the value of the one or more control parameters that is output by the control parameter value output unit and the virtual external force that is output by the virtual external force output unit are added together, and given to the control object.

### Claim 5 (depends on 1)

The reinforcement learning apparatus according to claim 1, further comprising a virtual external force approximator,
wherein the virtual external force approximator comprises:
a function approximator storage unit in which can be stored a function approximator including two or more third-type parameters serving as observable third-type parameters that are at least partially different from the second-type environment parameters and outputting a second virtual external force;
a state information obtaining unit that observes the control object, and obtains one or more pieces of state information relating to a state of the control object;
a second virtual external force calculation unit that calculates the second virtual external force by substituting the state information into the function approximator;
a difference information calculation unit that calculates difference information serving as information relating to a difference between the virtual external force that is output by the virtual external force generator and the second virtual external force; and
an approximation unit that updates a parameter of the function approximator so as to minimize the difference information.

### Claim 6 (depends on 5)

The reinforcement learning apparatus according to claim 5, further comprising:
a determination unit that determines whether the difference information satisfies a predetermined condition; and
a switching unit that switches the virtual external force that is output to the control object from an output of the virtual external force generator to an output of the virtual external force approximator if the determination unit has determined that the difference between the virtual external force that is output by the virtual external force generator and the second virtual external force is small enough to satisfy the predetermined condition.

### Claim 7 (depends on 1)

The reinforcement learning apparatus according to claim 1, wherein a two-point reaching movement task, which is a task of reaching a target point while avoiding an obstacle, is solved.

### Claim 8 (depends on 5)

The reinforcement learning apparatus according to claim 5,
wherein a coordinate system of the virtual external force is given by relative coordinates corresponding to a specific location of the control object.

### Claim 9 (depends on 5)

The reinforcement learning apparatus according to claim 5, further comprising a selection unit that obtains, from the one or more pieces of state information obtained by the state information obtaining unit, one or more pieces of state information having a change greater than a threshold,
wherein the second virtual external force calculation unit calculates the second virtual external force by substituting, into the function approximator, only the one or more pieces of state information obtained by the selection unit.

### Claim 10 (depends on 5)

A control apparatus that controls a moving control object using the virtual external force approximator according to claim 5 and the reinforcement learner according to claim 5.

### Claim 11

A reinforcement learning method implemented with a first-type environment parameter obtaining unit, a control parameter value calculation unit, a control parameter value output unit, a second-type environment parameter obtaining unit, a virtual external force calculation unit, and a virtual external force output unit, with a reward function for outputting a reward and a virtual external force function for outputting a virtual external force stored in a storage medium, the method comprising:
a first-type environment parameter obtaining step of obtaining, with the first-type environment parameter obtaining unit, a value of one or more first-type environment parameters relating to an environment of a moving control object;
a control parameter value calculation step of calculating, with the control parameter value calculation unit, a value of one or more control parameters maximizing a reward that is output by the reward function by substituting the value of the one or more first-type environment parameters and the virtual external force into the reward function;
a control parameter value output step of outputting, with the control parameter value output unit, the value of the one or more control parameters to the control object;
a second-type environment parameter obtaining step of obtaining, with the second-type environment parameter obtaining unit, a value of one or more second-type environment parameters relating to the virtual external force;
a virtual external force calculation step of calculating, with the virtual external force calculation unit, the virtual external force by substituting the value of the one or more second-type environment parameters into the virtual external force function; and
a virtual external force output step of outputting, with the virtual external force output unit, the virtual external force to the control object.
