Automated continuous validation for regulatory compliance of a computer system (CS) comprising a dynamic component

A system for automated continuous validation for regulatory compliance of a Computer System (CS) comprising a dynamic component, the system comprising:

• a memory storing instructions;
• one or more Input/Output (I/O) interfaces;
• and
• one or more hardware processors coupled to the memory via the one or more I/O interfaces, wherein the one or more hardware processors are configured by the instructions to:
• detect a learning of the dynamic component by analyzing system logs of the CS and generating a replica of the dynamic component of the CS deployed in production phase if the learning is detected;
• perform a User Acceptance Testing (UAT) in real time on the replica of the dynamic component for the detected learning by:
• a) creating a functional map by acquiring i) a manually approved regulatory document created for the dynamic component as per regulatory mandates or requirements from a static system to cater to an associated function, and ii) a user specification requirement document comprising user specification, functional specification, and model characteristics specification;
• b) generating, based on the manually approved regulatory document, a plurality of test entities comprising a plurality of test cases of varying types in accordance with a plurality of what-if scenarios, a plurality of test scripts for the plurality of test cases, and a synthetic test data;
• c) generating the synthetic test data for the plurality of test cases by:
• selecting a data sample from a plurality of zones defined by a pre-set variation in a plurality of values of a standard deviation (s) from a starting mean acceptance value (μx, μy) in a data sample space by recreating samples from the plurality of zones in accordance with each of the plurality of what-if scenarios of corresponding target outcome of the CS with the dynamic component to cover the plurality of test cases of the varying types, wherein the selected data sample is then mimicked in all quadrants formed around the starting mean acceptance value (μx, μy) to generate a complete polygram;
• selecting a base synthetic test data from the data sample by taking data that falls under curve of the complete polygram;
• and
• generating the synthetic test data by performing sub-sampling of the base synthetic test data by reducing plurality of parameters in the base synthetic test data using statistical models;
• d) setting an acceptance criterion for each of the plurality of test cases using three-sigma statistics defined by the standard deviation values (s, 2s, 3s) and a user specification requirement captured in the user specification requirement document;
• e) obtaining a standard Receiver Operator Characteristic (ROC) curve, from a ROC curve constructed for each of the plurality of test cases for the synthetic test data as input and corresponding outcome of the CS based on the set acceptance criterion for each of the plurality of test cases, wherein the ROC curve is constructed using the starting mean acceptance value, ending acceptance value derived from confidence interval defined for the standard deviation and a central intercept value;
• and
• f) checking whether the CS with the dynamic component meets a regulatory compliance of interest by executing the plurality of test scripts on the synthetic test data by:
• disabling the earning of the dynamic component so that the synthetic test data does not affect model characteristics of the dynamic component;
• determining whether a true outcome of the CS, with the learning of the dynamic component disabled, for corresponding input from a synthetic dataset, meets requirement of any variant of the standard ROC curve by checking a layout of data points on the standard ROC curve;
• and
• determining the CS as a complaint CS if all the data points of the true outcome fall under the standard ROC curve, otherwise declaring the CS as a non-complaint CS, wherein the determined learning of the dynamic component declared as non-compliant is rejected, and the dynamic component is rolled back to a previous version;
• and
• perform a base validation testing of the compliant CS to ensure repeatability, statistical stability, and consistency of the replica of the dynamic component before the learning is permitted to roll out in the production phase.

One or more non-transitory machine-readable information storage mediums comprising one or more instructions which when executed by one or more hardware processors cause:

• detecting learning of the dynamic component by analyzing system logs of the CS and generating a replica of the dynamic component of the CS deployed in production phase if the learning is detected;
• performing a User Acceptance Testing (UAT) in real time on the replica of the dynamic component for the detected learning by:
• a) creating a functional map by acquiring i) a manually approved regulatory document created for the dynamic component as per regulatory mandates or requirements from a static system to cater to an associated function, and ii) a user specification requirement document comprising user specification, functional specification, and model characteristics specification;
• b) generating, based on the manually approved regulatory document, a plurality of test entities comprising a plurality of test cases of varying types in accordance with a plurality of what-if scenarios, a plurality of test scripts for the plurality of test cases, and a synthetic test data;
• c) generating the synthetic test data for the plurality of test cases by:
• selecting a data sample from a plurality of zones defined by a pre-set variation in a plurality of values of a standard deviation (s) from a starting mean acceptance value (μx, μy) in a data sample space by recreating samples from the plurality of zones in accordance with each of the plurality of what-if scenarios of corresponding target outcome of the CS with the dynamic component to cover the plurality of test cases of the varying types, wherein the selected data sample is then mimicked in all quadrants formed around the starting mean acceptance value (μx, μy) to generate a complete polygram;
• selecting a base synthetic test data from the data sample by taking data that falls under curve of the complete polygram;
• and
• generating the synthetic test data by performing sub-sampling of the base synthetic test data by reducing plurality of parameters in the base synthetic test data using statistical models;
• d) setting an acceptance criterion for each of the plurality of test cases using three-sigma statistics defined by the standard deviation values (s, 2s, 3s) and a user specification requirement captured in the user specification requirement document;
• e) obtaining a standard Receiver Operator Characteristic (ROC) curve, from a ROC curve constructed for each of the plurality of test cases for the synthetic test data as input and corresponding outcome of the CS based on the set acceptance criterion for each of the plurality of test cases, wherein the ROC curve is constructed using the starting mean acceptance value, ending acceptance value derived from confidence interval defined for the standard deviation and a central intercept value;
• and
• f) checking whether the CS with the dynamic component meets a regulatory compliance of interest by executing the plurality of test scripts on the synthetic test data by:
• disabling the learning of the dynamic component so that the synthetic test data does not affect model characteristics of the dynamic component;
• determining whether a true outcome of the CS, with the learning of the dynamic component disabled, for corresponding input from a synthetic dataset, meets requirement of any variant of the standard ROC curve by checking a layout of data points on the standard ROC curve;
• and
• determining the CS as a complaint CS if all the data points of the true outcome fall under the standard ROC curve, otherwise declaring the CS as a non-complaint CS, wherein the determined learning of the dynamic component declared as non-compliant is rejected, and the dynamic component is rolled back to a previous version;
• and
• performing a base validation testing of the compliant CS to ensure repeatability, statistical stability, and consistency of the replica of the dynamic component before the learning is permitted to roll out in the production phase.