# US 2012/0084237 A1 · Data processing device, data processing method, and program

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2012/0084237 A1
- Document ID: US20120084237A1
- Status: Published application
- Kind: A1
- Application number: 13/248,296
- Filed: 2011-09-29
- Published: 2012-04-05
- Earliest claimed: 2010-10-04
- Assignees: Sony Corporation
- Inventors: Takashi HASUO; Kohtaro SABE; Kenta KAWAMOTO; Yukiko YOSHIKE
- Web page: https://patentgrep.com/patent/US20120084237A1

## CPC

- G06N 3/006: based on simulated virtual individual or collective life forms, e.g. social simulations or particle swarm optimisation [PSO]
- G06N 20/00: Machine learning
- G06N 20/10: using kernel methods, e.g. support vector machines [SVM]
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

A data processing device includes a state value calculation unit which calculates a state value of which the value increases as much as a state with a high transition probability for each state of the state transition model, an action value calculation unit which calculates an action value, of which the value increases as a transition probability increases for each state of the state transition model and each action that the agent can perform, a target state setting unit which sets a state with great unevenness in the action value among states of the state transition model to a target state that is the target to reach by action performed by the agent, and an action selection unit which selects an action of the agent so as to move toward the target state.

## Claims

### Claim 1

A data processing device comprising:
a state value calculation unit which calculates a state value having a predetermined state of a state transition model, in which a state is transited by an action performed by an agent that can act, set as a reference, of which the value increases as much as a state with a high transition probability to a state close to the predetermined state, for each state of the state transition model based on the state transition model of each action;
an action value calculation unit which calculates an action value, of which the value increases as a transition probability to a state with a high state value having the predetermined state set as a reference increases, for each state of the state transition model and each action that the agent can perform, based on the state transition model and the state value having the predetermined state set as a reference;
a target state setting unit which sets a state with great unevenness in the action value among states of the state transition model to a target state that is the target to reach by an action performed by the agent, based on the action value; and
an action selection unit which selects an action of the agent so as to move toward the target state.

### Claim 2 (depends on 1)

The data processing device according to claim 1, further comprising:
a state recognition unit which recognizes the current state which is a state where an observation value, which is observed by the agent from the outside, is observed among states of the state transition model based on the observation value,
wherein the predetermined state is the current state, and
wherein the state value calculation unit calculates a state value having the current state set as a reference, of which the value increases as much as a state with a high transition probability to a state close to the current state.

### Claim 3 (depends on 2)

The data processing device according to claim 2, wherein the action selection unit calculates a state value having the target state set as a reference, of which the value increases as a state with a high transition probability to a state close to the target state, for each state of the state transition model based on the state transition model, calculates an action value of which the value increases as high as a transition probability to a state with a high state value having the target state set as a reference for each state of the state transition model and each action that the agent can perform based on the state transition model and the state value having the target state set as a reference, and selects an action of the agent so as to move toward the target state based on an action value of the current state.

### Claim 4 (depends on 3)

The data processing device according to claim 3, further comprising:
a model updating unit which updates a state transition model for an action of the agent, in which state transition to the current state occurs, based on the state transition to the current state.

### Claim 5 (depends on 4)

The data processing device according to claim 4,
wherein the state transition model for a predetermined action indicates a frequency of transition to a second state by the predetermined action by the agent in a first state, and
wherein the model updating unit updates the state transition model by increasing the frequency.

### Claim 6 (depends on 5)

The data processing device according to claim 5,
wherein the agent acts in an action environment where the agent acts, assuming a predetermined space as the action environment, and observes a position of the agent in the action environment as the observation value, and
wherein the state indicates a small area obtained by dividing the action environment into such small areas.

### Claim 7 (depends on 6)

The data processing device according to claim 6, wherein the action selection unit determines whether or not the current state coincides with the target state, and selects an action of the agent so as to move toward the target state based on an action value of the current state when the current state does not coincide with the target state.

### Claim 8 (depends on 7)

The data processing device according to claim 7,
wherein, when the current state coincides with the target state, the state value calculation unit re-calculates a state value having the current state set as a reference based on the state transition model, the action value calculation unit re-calculates the action value based on the state transition model and the state value having the current state set as a reference, and the target state setting unit re-sets the target state based on the action value.

### Claim 9 (depends on 2)

The data processing device according to claim 2, wherein the target state setting unit obtains a variance of the action value for each state of the state transition model, and sets a state to be reached from the current state by state transitions within a predetermined number of times among states, in which a variance of the action value is equal to or higher than a predetermined threshold value, to the target state.

### Claim 10 (depends on 3)

The data processing device according to claim 3, wherein the action selection unit selects an action of the agent so as to move toward the target state based on the action value of the current state with a 6-greedy method or a softmax method.

### Claim 11

A data processing method of a data processing device, comprising:
calculating a state value having a predetermined state of a state transition model, in which a state is transited by an action performed by an agent that can act, set as a reference, of which the value increases as much as a state with a high transition probability to a state close to the predetermined state, for each state of the state transition model based on the state transition model of each action;
calculating an action value, of which the value increases as a transition probability to a state with a high state value having the predetermined state set as a reference increases, for each state of the state transition model and each action that the agent can perform, based on the state transition model and the state value having the predetermined state set as a reference;
setting a state with great unevenness in the action value among states of the state transition model to a target state that is the target to reach by an action performed by the agent, based on the action value; and
selecting an action of the agent so as to move toward the target state.

### Claim 12

A program causing a computer to function as:
a state value calculation unit which calculates a state value having a predetermined state of a state transition model, in which a state is transited by an action performed by an agent that can act, set as a reference, of which the value increases as much as a state with a high transition probability to a state close to the predetermined state, for each state of the state transition model based on the state transition model of each action;
an action value calculation unit which calculates an action value, of which the value increases as a transition probability to a state with a high state value having the predetermined state set as a reference increases, for each state of the state transition model and each action that the agent can perform, based on the state transition model and the state value having the predetermined state set as a reference;
a target state setting unit which sets a state with great unevenness in the action value among states of the state transition model to a target state that is the target to reach by action performed by the agent, based on the action value; and
an action selection unit which selects an action of the agent so as to move toward the target state.
