# US 10,871,753 B2 · Feedback loop driven end-to-end state control of complex data-analytic systems > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,871,753 B2 - Document ID: US10871753B2 - Status: Granted - Kind: B2 - Application number: 15/445,077 - Filed: 2017-02-28 - Published: 2018-02-01 - Granted: 2020-12-22 - Earliest claimed: 2016-07-27 - Assignees: Accenture Global Solutions Limited - Applicants: Accenture Global Solutions Limited - Inventors: Jaeyoung Christopher Kang; Jürgen Albert Weichenberger; Teresa Sheausan Tung; William R. Gatehouse; Tiffany Cecilia Dharma; Jan Andre Nicholls - Web page: https://patentgrep.com/patent/US10871753B2 ## CPC - G05B 17/02: electric - B01D 19/0063: {Regulation, control including valves and floats (for construction and details of valves F16K)} - B01D 19/04: by addition of chemical substances - G05B 13/021: {in which a variable is automatically adjusted to optimise the performance} - G05B 13/026: {using a predictor} - G05B 13/04: involving the use of models or simulators - G06N 20/00: Machine learning - G05B: CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS (systems for controlling or regulating non-electric variables G05D; systems for regulating electric or magnetic variables G05F; control devices or systems insofar as characterised by mechanical features only G05G) - B01D: SEPARATION (separating solids from solids by wet methods B03B, B03D; by pneumatic jigs or tables B03B; by other dry methods B07; magnetic or electrostatic separation of solid materials from solid materials or fluids, separation by high-voltage electric fields B03C; centrifuges, vortex apparatus B04; presses per se for squeezing-out liquid from liquid-containing material B30B9/02; treatment of water C02F, e.g. softening by ion-exchange C02F1/42; {arrangements of air intake cleaners in gas turbine plants F02C7/05} ; arrangements or mounting of filters in air-conditioning, air-humidification or ventilation F24F13/28) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS ## Abstract A system provides feedback driven end-to-end state control of a data model. A data model may be used to model the behavior of a monitored system to predict future events. The system may be used to ensure proper operation of the data model. Contingency data models may be executed when a failure is detected. Further, when the system detects accuracy that is out of tolerance, the system may initiate retraining of the data model being currently used. ## Claims ### Claim 1 A method comprising: accessing, in validation memory, a dependency matrix for a child data model; responsive to the dependency matrix for the child data model, determining a parent data model that is mandatory for execution of the child data model; comparing, using model execution circuitry, feedback data generated using the parent data model to a template to determine whether the feedback data conforms to the template; when the feedback data does not conform: cancelling an execution of the child data model; generating a first prompt indicating that execution of the child data model has been cancelled; determining a contingency data model for the parent data model, the child data model or both; executing the contingency data model; and passing contingency output from the contingency data model to accuracy analysis circuitry; using the accuracy analysis circuitry, performing a first accuracy check to determine whether to retrain the contingency data model, the first accuracy check comprising a comparison of a predicted value from the contingency output to a later-measured physical sensor value; when the accuracy analysis circuitry determines to retrain the contingency data model: via the model execution circuitry, retraining by adjusting an first operational parameter for the contingency data model; and after retraining, re-launching the contingency data model using the model execution circuitry; and when the feedback data conforms, execute the child data model. ### Claim 2 (depends on 1) The method of claim 1, where: the dependency matrix further includes an entry indicating an non-mandatory criterion for execution of the child model; and the method further includes: when the non-mandatory criterion is not met and the parent model has not failed: generating a second prompt indicating that the non-mandatory criterion has not been met; and executing the child data model. ### Claim 3 (depends on 2) The method of claim 2, where generating the second prompt comprises presenting a user interface option to cancel execution of the child data model. ### Claim 4 (depends on 1) The method of claim 1, where the template comprises an indication of an output data format for the parent data model. ### Claim 5 (depends on 4) The method of claim 4, where the method further comprises: after comparing the feedback data to the template; determining whether a conversion function is available to convert the feedback data to the output data format; when the conversion function is available, determining that the parent model has not failed; and when the conversion function is not available, determining that the parent model has failed. ### Claim 6 (depends on 1) The method of claim 1, where executing the contingency data model comprises executing a data model calibrated for lower predictive accuracy than the child data model. ### Claim 7 (depends on 1) The method of claim 1, where the method further comprises: generating model output using the child data model; passing the model output to the accuracy analysis circuitry; using the accuracy analysis circuitry, performing a second accuracy check to determine whether to retrain the child data model; when the accuracy analysis circuitry determines to retrain the child data model: retraining the child data model by adjusting a second operational parameter for the child data model; and after retraining, re-launch the child data model. ### Claim 8 (depends on 1) The method of claim 1, further comprising, after executing the child data model, determining the retrain the child data model after a pre-defined interval has passed. ### Claim 9 (depends on 1) The method of claim 1, further comprising, generating, via deployment execution circuitry, the dependency matrix for the child data model. ### Claim 10 (depends on 9) The method of claim 9, where generating the dependency matrix comprises including indications whether individual ones of criteria listed within the dependency matrix comprise mandatory criteria. ### Claim 11 (depends on 1) The method of claim 1, where performing the first accuracy check comprises applying a confusion matrix analysis, the confusion matrix analysis comprising: generating a first memory entry within a confusion matrix accounting for false negative predictions based on the contingency output; and generating a second memory entry within the confusion matrix accounting for false positive predictions based on the contingency output. ### Claim 12 (depends on 1) The method of claim 1, where performing the first accuracy check comprises applying a receiver operating characteristic (ROC) curve analysis, the ROC curve analysis comprising determining an area under an ROC curve representation of the contingency output to select a ratio between false negative predictions and false positive predictions. ### Claim 13 A system comprising: validation memory comprising a dependency matrix for a child data model, the dependency matrix including an indicator of a parent data model that is mandatory for execution of the child data model; accuracy analysis circuitry configured to perform an accuracy check determine whether to retrain the child data model, the accuracy check comprising a comparison of a predicted value from model output to later-measured physical sensor value; and model execution circuitry in data communication with the validation memory, the model execution circuitry configured to: access the dependency matrix for the child data model; responsive to the dependency matrix for the child data model, determine that the parent data model is mandatory for execution of the child data model; compare feedback data generated using the parent data model to a template to determine whether the parent data model has failed; when the parent model has failed: cancel an execution of the child model; generate a prompt indicating that execution of the child model has been cancelled; determine a contingency data model for the parent data model, the child data model or both; and execute the contingency data model; when the parent model has not failed: execute the child data model; generate the model output using the child data model; pass the model output to the accuracy analysis circuitry; when the accuracy analysis circuitry determines to retrain the child data model: retrain the child data model by adjusting an operational parameter for the child data model; and after retraining, re-launch the child data model. ### Claim 14 (depends on 13) The system of claim 13, further comprising deployment execution circuitry in data communication with the validation memory, the deployment execution circuitry configured to: generate the dependency matrix; and cause the validation memory to store the dependency matrix. ### Claim 15 (depends on 14) The system of claim 14, where the deployment execution circuitry is configured to generate the dependency matrix by indicating whether multiple criteria included within the dependency matrix comprise mandatory criteria for execution of the child data model.