# US 2013/0085974 A1 · Using cyclic markov decision process to determine optimum policy

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2013/0085974 A1
- Document ID: US20130085974A1
- Status: Published application
- Kind: A1
- Application number: 13/589,407
- Filed: 2012-08-20
- Published: 2013-04-04
- Earliest claimed: 2011-09-30
- Assignees: International Business Machines Corporation
- Inventors: Takayuki Osogami; Raymond H. Rudy
- Web page: https://patentgrep.com/patent/US20130085974A1

## CPC

- G06N 7/01: Probabilistic graphical models, e.g. probabilistic networks
- G06N 20/00: Machine learning
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

A method for determining an optimum policy by using a Markov decision process in which T subspaces each have at least one state having a cyclic structure includes identifying, with a processor, subspaces that are part of a state space; selecting a t-th (t is a natural number, t≦T) subspace among the identified subspaces; computing a probability of, and an expected value of a cost of, reaching from one or more states in the selected t-th subspace to one or more states in the t-th subspace in a following cycle; and recursively computing a value and an expected value of a cost based on the computed probability and expected value of the cost, in a sequential manner starting from a (t −1)-th subspace.

## Claims

### Claim 1

A method for determining an optimum policy by using a Markov decision process in which T subspaces each have at least one state having a cyclic structure, comprising:
identifying, with a processor, subspaces that are part of a state space;
selecting a t-th (t is a natural number, t≦T) subspace among the identified subspaces;
computing a probability of, and an expected value of a cost of, reaching from one or more states in the selected t-th subspace to one or more states in the t-th subspace in a following cycle; and
recursively computing a value and an expected value of a cost based on the computed probability and expected value of the cost, in a sequential manner starting from a (t−1)-th subspace.

### Claim 2 (depends on 1)

The method according to claim 1, wherein selection of a subspace having fewest states among the T subspaces is received as the t-th subspace.

### Claim 3 (depends on 1)

The method according to claim 1, wherein an average value of values and an average value of expected values of cost of one or more states in the t-th subspace are computed.

### Claim 4 (depends on 1)

The method according to claim 1, wherein a value variable for each of the T subspaces is computed to optimize the Markov decision process.
