# US 2004/0064735 A1 · Control systems and methods using a partially-observable markov decision process (PO-MDP)

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2004/0064735 A1
- Document ID: US20040064735A1
- Status: Published application
- Kind: A1
- Application number: 10/638,818
- Filed: 2003-08-11
- Published: 2004-04-01
- Earliest claimed: 2002-08-09
- Assignees: BAE Systems Advanced Information Technologies Inc.
- Inventors: Tiffany M. Frazier; O. Patrick Kreidl
- Web page: https://patentgrep.com/patent/US20040064735A1

## CPC

- G06F 21/316: {by observing the pattern of computer usage, e.g. typical user behaviour}
- G06F 21/55: Detecting local intrusion or implementing counter-measures
- G06F 21/552: {involving long-term monitoring or reporting}
- G06F 21/554: {involving event detection and direct action}
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

Methods and systems for controlling at least a part of a microprocessor system, that include, based at least in part on objectives of at least one electronic attack, using a partially observable Markov decision process (PO-MDP) to provide a model of at least part of the microprocessor system, the PO-MDP including a controller, the controller including an estimation policy to recursively generate a state estimate at stage k based on a state estimate at stage k−1, a control at stage k−1, and an observation at stage k, and, a response policy to select a control at stage k based on the state estimate at stage k, and cause the selected control to be provided to at least a part of the microprocessor system.

## Claims

### Claim 1

A method for controlling at least a part of a microprocessor system, the method comprising:
based at least in part on objectives of at least one electronic attack, using a partially observable Markov decision process (PO-MDP) to provide a model of at least part of the microprocessor system, the PO-MDP including a controller, the controller comprising:
an estimation policy to recursively generate a state estimate at stage k based on a state estimate at stage k−1, a control at stage k−1, and an observation at stage k, and,
a response policy to select a control at stage k based on the state estimate at stage k, and cause the selected control to be provided to at least a part of the microprocessor system.

### Claim 2 (depends on 1)

A method according to claim 1, where using the PO-MDP to model includes collecting at least one training set based on a sample electronic attack.

### Claim 3 (depends on 2)

A method according to claim 2, where collecting includes collecting from one or more sensors, where the one or more sensors are in communications with at least part of the microprocessor system.

### Claim 4 (depends on 3)

A method according to claim 3, where collecting includes determining sensor observation probabilities.

### Claim 5 (depends on 1)

A method according to claim 1, further comprising selecting the estimation policy and the response policy to minimize an average cost per stage.

### Claim 6 (depends on 5)

A method according to claim 5, where selecting the response policy includes selecting based on at least one of: an infinite-horizon assumption and a multi-state look-ahead optimization technique.

### Claim 7 (depends on 5)

A method according to claim 5, where selecting the estimation policy includes constructing a recursive estimator to provide a sufficient statistical state estimate.

### Claim 8 (depends on 1)

A method according to claim 1, where using includes:
selecting the estimation policy based on minimizing an error per stage.

### Claim 9 (depends on 8)

A method according to claim 8, where minimizing the error per stage includes determining a distance metric.

### Claim 10 (depends on 9)

A method according to claim 9, where determining the distance metric includes determining a difference between an estimate of each state of the model, and the true state.

### Claim 11 (depends on 9)

A method according to claim 9, where determining the distance metric includes computing a Euclidean norm.

### Claim 12 (depends on 8)

A method according to claim 8, further comprising:
selecting the response policy based on a fixed estimation policy and minimizing the average cost per stage.

### Claim 13 (depends on 12)

A method according to claim 12, where minimizing the average cost per stage includes determining a minimum cost-to-go based on the state estimate at stage k.

### Claim 14 (depends on 1)

A method according to claim 1, further comprising:
receiving observations from one or more sensors at stage k, and, associating the observations with observation probabilities based on the state estimate at stage k and the control at stage k−1.

### Claim 15 (depends on 14)

A method according to claim 14, where receiving observations includes producing a distance measure based on the received observations.

### Claim 16 (depends on 14)

A method according to claim 14, where receiving observations include applying the received observations to a detection scheme.

### Claim 17 (depends on 16)

A method according to claim 16, where the detection scheme includes employing a Bayesian classifier.

### Claim 18 (depends on 1)

A method according to claim 1, further comprising specifying at least one of: a state space associated with the model, a control space associated with the model, and an observation space associated with the model.

### Claim 19 (depends on 1)

A method according to claim 1, further comprising specifying one or more state transition probabilities for a state at stage k+1 based on the state at stage k and a control at stage k.

### Claim 20 (depends on 1)

A method according to claim 1, further comprising specifying one or more sensor observation probabilities for stage k based on a state at stage k and a control at stage k−1.

### Claim 21 (depends on 1)

A method according to claim 1, where the model includes states of at least part of at least one of an electronic intrusion and an electronic attack based on at least a part of at least one of: a microprocessor-controlled device and a set of instructions.

### Claim 22 (depends on 1)

A method according to claim 1, where the model is a stationary model.

### Claim 23

A system for controlling at least a part of a microprocessor device, the system comprising:
a partially observable Markov decision process (PO-MDP) controller, the PO-MDP controller comprising:
two or more states based on one or more objectives of an electronic attack,
one or more state transition probabilities based on a state at a stage k and a control at a stage k,
one or more sensor observation probabilities based on the state at stage k and a control at a stage k−1,
one or more state transition costs based on the state at stage k, the control at stage k−1, and a state estimate for stage k+1,
a recursive estimator to generate a probabilistic state estimate for stage k, and,
a response policy to associate the probabilistic state estimate to a control for stage k.

### Claim 24 (depends on 23)

A system according to claim 23, where the recursive estimator computes a sufficient statistic.

### Claim 25 (depends on 23)

A system according to claim 23, where the probabilistic state estimate is a sufficient statistic.

### Claim 26 (depends on 23)

A system according to claim 23, further comprising at least one actuator to affect the control for stage k.

### Claim 27 (depends on 23)

A system according to claim 23, further comprising one or more sensors to provide one or more sensor observations based on the system.

### Claim 28 (depends on 27)

A system according to claim 27, where the one or more sensor observations include sensor observations based on at least one of: a process activity associated with the system, a network activity associated with the system, and a kernel activity associated with the system.

### Claim 29 (depends on 23)

A system according to claim 23, where the system includes at least a part of at least one of: a microprocessor-controlled device and a set of instructions.

### Claim 30 (depends on 23)

A system according to claim 23, where the system states include one or more states associated with at least one of an electronic attack and an electronic intrusion.

### Claim 31 (depends on 23)

A system according to claim 23, where the control for stage k includes at least one of a default process and a kill process.
