# US 9,753,441 B2 · Controlling dynamical systems with bounded probability of failure > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 9,753,441 B2 - Document ID: US9753441B2 - Status: Granted - Kind: B2 - Application number: 14/275,933 - Filed: 2014-05-13 - Published: 2014-11-13 - Granted: 2017-09-05 - Earliest claimed: 2013-05-13 - Assignees: Massachusetts Institute of Technology - Applicants: Massachusetts Institute of Technology - Inventors: Vu Anh Huynh; Emilio Frazzoli - Web page: https://patentgrep.com/patent/US9753441B2 ## CPC - G05B 13/04: involving the use of models or simulators - G05B 23/0248: {Causal models, e.g. fault tree; digraphs; qualitative physics} - G05B: CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS (systems for controlling or regulating non-electric variables G05D; systems for regulating electric or magnetic variables G05F; control devices or systems insofar as characterised by mechanical features only G05G) ## Abstract A computer-based method controls a dynamical system in an uncertain environment within a bounded probability of failure. The dynamical system has a state space and a control space. The method includes diffusing a risk constraint corresponding to the bounded probability of failure into a martingale that represents a level of risk tolerance associated with the dynamical system over time. The state space and the control space of the dynamical system are augmented with the martingale to create an augmented model with an augmented state space and an augmented control space. The method may include iteratively constructing one or more Markov Decision Processes (MDPs), with each iterative MDP represents an incrementally refined model of the dynamical system. The method further includes computing a first solution based on the augmented model or, if additional time was available, based on one of the MDP iterations. ## Claims ### Claim 1 A computer-implemented method for controlling a dynamical system in an uncertain environment within a bounded probability of failure, wherein the dynamical system has a state space and a control space, the method comprising: diffusing, with at least one computer-based processor, a risk constraint that corresponds to the bounded probability of failure associated with the dynamical system into a martingale that represents a level of risk tolerance associated with the dynamical system over time; augmenting the state space and the control space of the dynamical system with the martingale to create an augmented model for the dynamical system, wherein the augmented model has an augmented state space and an augmented control space; if additional time is available before a control signal needs to be returned to the dynamical system, then iteratively constructing one or more Markov Decision Processes (MDPs), wherein each iterative MDP represents an incrementally refined model of the dynamical system, relative to the augmented model and any previously-constructed MDP iteration; and computing a first solution based on the augmented model or, if additional time was available, based on one of the MDP iterations. ### Claim 2 (depends on 1) The computer-implemented method of claim 1, further comprising returning a first control signal to the dynamical system based on the computed first solution. ### Claim 3 (depends on 2) The computer-implemented method of claim 2, further comprising: after returning the control signal to the dynamical system based on the computed solution, continuing to iteratively construct one or more additional MDPs, wherein each additional MDP represents a further incrementally refined model of the dynamical system, relative to one or more previously-constructed MDP iterations; and computing a second solution based on one of the additional MDP iterations. ### Claim 4 (depends on 3) The computer-implemented method of claim 3, further comprising returning a second control signal to the dynamical system based on the computed second solution. ### Claim 5 (depends on 4) The computer-implemented method of claim 4, further comprising: controlling the dynamical system based on the computed first signal; and subsequently, controlling the dynamical system based on the computed second signal. ### Claim 6 (depends on 1) The computer-implemented method of claim 1, wherein creating the augmented model transforms controlling the dynamical system from a risk-constrained problem into a stochastic target problem. ### Claim 7 (depends on 1) The computer-implemented method of claim 1, wherein each one of the MDP iterations has an augmented state space and an augmented control space. ### Claim 8 (depends on 1) The computer-implemented method of claim 1, wherein computing the first solution utilizes a rapidly-exploring sampling technique and approximating Markov chains. ### Claim 9 (depends on 1) The computer-implemented method of claim 1, wherein constructing the MDP at each one of the iterations comprises: sampling from an augmented state space associated with the augmented model or a previously-constructed one of the MDP iterations; and computing one or more boundary values based on the sampling. ### Claim 10 (depends on 1) The computer-implemented method of claim 1, wherein computing the first solution based on one of the MDP iterations comprises computing the first solution based on a latest one of the MDP iterations. ### Claim 11 (depends on 1) The computer-implemented method of claim 1, further comprising enabling a human to enter the risk constraint, via a computer-based data input device, associated with the dynamical system. ### Claim 12 (depends on 1) The computer-implemented method of claim 1, further comprising: enabling a human to identify, through a computer-based data input device, a desired final state associated with the dynamical system; and using the identified, desired final state in the controlling of the dynamical system. ### Claim 13 A computer-based system comprising: a controller for a dynamical system to operate in an uncertain environment within a bounded probability of failure, wherein the dynamical system has a state space and a control space; at least one computer-based processor configured to: diffuse a risk constraint associated with the dynamical system that corresponds to the bounded probability of failure into a martingale that represents a level of risk tolerance associated with the dynamical system over time; augment the state space and the control space of the dynamical system with the martingale to create an augmented model for the dynamical system, wherein the augmented model has an augmented state space and an augmented control space; if additional time is available before a control signal needs to be returned to the dynamical system, then iteratively construct one or more Markov Decision Processes (MDPs), wherein each iterative MDP represents an incrementally refined model of the dynamical system, relative to the augmented model and any previously-constructed MDP iteration; and compute a first solution based on the augmented model or, if additional time was available, based on one of the MDP iterations. ### Claim 14 (depends on 13) The computer-based system of claim 13, wherein the at least one computer-based processor is further configured to return a first control signal to the dynamical system based on the computed first solution. ### Claim 15 (depends on 14) The computer-based system of claim 14, wherein the at least one computer-based processor is further configured to: after returning the control signal to the dynamical system based on the computed solution, continue to iteratively construct one or more additional MDPs, wherein each additional MDP represents a further incrementally refined model of the dynamical system, relative to one or more previously-constructed MDP iterations; and compute a second solution based on one of the additional MDP iterations. ### Claim 16 (depends on 15) The computer-based system of claim 15, wherein the at least one computer-based processor is further configured to return a second control signal to the dynamical system based on the computed second solution. ### Claim 17 (depends on 15) The computer-based system of claim 15, wherein the computer-based controller is configured to: control the dynamical system based on the computed first signal; and subsequently, control the dynamical system based on the computed second signal. ### Claim 18 (depends on 13) The computer-based system of claim 13, wherein each one of the MDP iterations has an augmented state space and an augmented control space. ### Claim 19 (depends on 13) The computer-based system of claim 13, wherein the at least one computer-based processor is configured to compute the first solution by utilizing a rapidly-exploring sampling technique and approximating Markov chains. ### Claim 20 (depends on 13) The computer-based system of claim 13, wherein the at least one computer-based processor is configured to construct the MDP at each one of the iterations by: sampling from an augmented state space associated with the augmented model or a previously-constructed one of the MDP iterations; and computing one or more boundary values based on the sampling. ### Claim 21 (depends on 13) The computer-based system of claim 13, further comprising: a computer-based data input device coupled to the at least one computer-based processor and/or the computer-based controller. ### Claim 22 (depends on 21) The computer-based system of claim 21, wherein the computer-based data input device is configured to enable a human to enter the risk constraint associated with the dynamical system and/or to identify a desired final state associated with the dynamical system. ### Claim 23 (depends on 22) The computer-based system of claim 22, wherein the at least one computer-based processor is configured to use the identified, desired final state in the controlling of the dynamical system. ### Claim 24 A non-transitory, computer-readable medium that stores instructions executable by at least one computer-based processor to perform the operations comprising: diffusing a risk constraint associated with a dynamical system that corresponds to a bounded probability of failure into a martingale that represents a level of risk tolerance associated with the dynamical system over time; augmenting a state space and a control space of the dynamical system with the martingale to create an augmented model for the dynamical system, wherein the augmented model has an augmented state space and an augmented control space; if additional time is available before a control signal needs to be returned to the dynamical system, then iteratively constructing one or more Markov Decision Processes (MDPs), wherein each iterative MDP represents an incrementally refined model of the dynamical system, relative to the augmented model and any previously-constructed MDP iteration; and computing a first solution based on the augmented model or, if additional time was available, based on one of the MDP iterations.