# US 2014/0156031 A1 · Adaptive Stochastic Controller for Dynamic Treatment of Cyber-Physical Systems

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2014/0156031 A1
- Document ID: US20140156031A1
- Status: Published application
- Kind: A1
- Application number: 14/177,020
- Filed: 2014-02-10
- Published: 2014-06-05
- Earliest claimed: 2011-08-11
- Assignees: The Trustees of Columbia University in the City of New York
- Applicants: The Trustees of Columbia University in the city of New York
- Inventors: Roger N. Anderson; Albert Boulanger; Leon L. Wu; Kevin Mclnerney; Timothy Teravainen; Bibhas Chakraborty
- Web page: https://patentgrep.com/patent/US20140156031A1

## CPC

- G05B 13/04: involving the use of models or simulators
- G05B 19/4184: {characterised by fault tolerance, reliability of production system}
- G06Q 10/06: Resources, workflows, human or project management; Enterprise or organisation planning; Enterprise or organisation modelling
- G06Q 50/06: Energy or water supply
- G06F 2111/08: Probabilistic or stochastic CAD
- Y02P 90/02: Total factory control, e.g. smart factories, flexible manufacturing systems [FMS] or integrated manufacturing systems [IMS]
- G05B: CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS (systems for controlling or regulating non-electric variables G05D; systems for regulating electric or magnetic variables G05F; control devices or systems insofar as characterised by mechanical features only G05G)
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- Y02P: CLIMATE CHANGE MITIGATION TECHNOLOGIES IN THE PRODUCTION OR PROCESSING OF GOODS

## Abstract

Techniques for generating a dynamic treatment control policy for a cyber-physical system having one or more components, including a data collector for collecting data representative of the cyber-physical system, and adaptive stochastic controller including one or more models for generating a predicted value corresponding to available actions based on an objective function, and an approximate dynamic programming element configured to receive actual operation metrics corresponding to the available actions. The approximate dynamic programming element can learn a state-action map and generate a dynamic treatment control policy using the one or more models.

## Claims

### Claim 1

A system for generating a dynamic treatment control policy for a cyber-physical system having one or more components, comprising:
a data collector to collect data representative of the cyber physical system; and
an adaptive stochastic controller operatively coupled to the data collector and adapted to receive collected data therefrom, the adaptive stochastic controller comprising:
one or more models for generating a predicted value corresponding to one or more available actions based on an objective function;
an approximate dynamic programming element, configured to receive one or more actual operation metrics corresponding to one of the available actions and to learn a state-action map; and
wherein the adaptive stochastic controller is adapted to generate a dynamic treatment control policy using the one or more models.

### Claim 2 (depends on 1)

The system of claim 1, wherein the approximate dynamic programming element is further configured to adjust the one or more models using the actual operation metrics.

### Claim 3 (depends on 1)

The system of claim 1, wherein the data collector includes a receiver for receiving outage derived data sets and the collected data includes one or more of static data about the components, dynamic external data, and dynamic data about the components.

### Claim 4 (depends on 1)

The system of claim 1, wherein the objective function comprises a mean time between failure, and wherein the one or more models for generating a predicted value includes a model for generating a predicted mean time between failure for each component of the cyber-physical system.

### Claim 5 (depends on 4)

The system of claim 4, wherein the model is further configured to calculate a predicted metric based on a gradient of a failure rate for each component, the predicted metric being the difference in failure rate.

### Claim 6 (depends on 4)

The system of claim 4, wherein the model is further configured to calculate the predicted mean time between failure based on a semiparametric model.

### Claim 7 (depends on 1)

The system of claim 1, wherein the one or more models further includes a propensity model for inversely weighting components with a history of independent treatment using the data representative of the cyber-physical system.

### Claim 8 (depends on 1)

The system of claim 1, wherein the one or more models further includes a learning system for learning behavior of the cyber-physical system and learn a state-action map and optionally adjusting the one or more models using the one or more actual operation metrics.

### Claim 9 (depends on 1)

The system of claim 1, wherein the approximate dynamic programming element is further configured to learn the state-action map and optionally adjust the one or more models using Q-learning for comparison of the predicted value and the one or more actual operation metrics.

### Claim 10 (depends on 1)

The system of claim 1, wherein the available actions is selected from the group consisting of repairing, replacing, or delaying repairing or replacing one or more components of the cyber-physical system.

### Claim 11 (depends on 1)

The system of claim 1, wherein the available actions is selected from the group consisting of repairing, replacing, or deciding not to repair or replace one or more components of the cyber-physical system in an order determined by the one or more models.

### Claim 12 (depends on 1)

The system of claim 1, wherein the actual operation metrics is selected from the group consisting of recorded data of the components, estimated data of the components, and external data.

### Claim 13

A method for generating a dynamic treatment control policy for a cyber-physical system having one or more components, where one or more available actions on each of the one or more components can be taken, comprising:
generating a predicted value corresponding to the one or more available actions based on an objective function and using one or more models using data representative of the cyber-physical system;
receiving one or more actual operation metrics at an approximate dynamic programming element, the actual operation metrics corresponding one or more executed actions, the one or more executed actions corresponding to one of the one or more available actions;
learning a state-action map with the approximate dynamic programming element using the actual operation metrics; and
generating a dynamic treatment control policy using the predicted value and the one or more models.

### Claim 14 (depends on 13)

The method of claim 13, further comprising adjusting the one or more models with the approximate dynamic programming element using the actual operation metrics; and

### Claim 15 (depends on 13)

The method of claim 13, further comprising transmitting the data representative of the cyber-physical system from an outage derived data set to the one or more models, and wherein the data representative of the cyber-physical system includes one or more of static data about the components, dynamic external data, and dynamic data about the components.

### Claim 16 (depends on 13)

The method of claim 13, wherein the objective function comprises a mean time between failure and wherein generating a predicted value includes using a model generating a predicted mean time between failure for each component of the cyber-physical system.

### Claim 17 (depends on 16)

The method of claim 16, wherein using one or more models further comprises using a model configured to calculate a predicted metric based on a gradient of a failure rate for each component, the predicted metric being the difference in failure rate.

### Claim 18 (depends on 16)

The method of claim 16, wherein using one or more models further comprises using a model configured to calculate the predicted mean time between failure based on a semiparametric model.

### Claim 19 (depends on 13)

The method of claim 13, wherein generating a predictive value further comprises inversely weighting components with a history of independent treatment using the data representative of the cyber-physical system.

### Claim 20 (depends on 13)

The method of claim 13, wherein adjusting the one or more models further comprises learning a behavior of the cyber-physical system, learning the state-action maps, and optionally adjusting using the behavior and the one or more actual operation metrics.

### Claim 21 (depends on 13)

The method of claim 13, wherein the adjusting further includes learning the state-action map and optionally adjusting the one or more models using Q-learning for comparison of the predicted value and the one or more actual operation metrics.

### Claim 22 (depends on 13)

The method of claim 13, wherein the one or more available options is selected from the group consisting of repairing, replacing, or delaying repairing or replacing one or more components of the cyber-physical system, and further comprising executing the one or more available options.

### Claim 23 (depends on 13)

The method of claim 13, wherein the actual operation metrics is selected from the group consisting of the components, estimated data of the components, and external data, and further comprising:
collecting the actual operation metrics; and
transmitting the actual operation metrics to the approximate dynamic programming element.