Stress test impact isolation and mapping

A system comprising:

• a memory device;
• and
• one or more processing units coupled with the memory device, the one or more processing units configured to perform a method comprising:
• generating a plurality of test cases for testing a system under test (SUT) that is being executed in an active environment, the plurality of test cases generated based on a coverage model of the SUT, wherein the coverage model comprises a plurality of attributes;
• recording a first difference vector by executing the plurality of test cases on a baseline system, which is different from the SUT, and monitoring performance parameters of the baseline system before and after executing the plurality of test cases, wherein the first difference vector comprises a plurality of performance records corresponding respectively to the test cases, wherein each performance record comprises differences in the performance parameters of the baseline system from before and after the execution of a corresponding test case, the performance parameters comprising resource usage metrics of each of the plurality of test cases;
• recording a second difference vector by executing the plurality of test cases on the SUT and monitoring performance parameters of the SUT before and after executing the plurality of test cases, wherein the second difference vector also comprises a plurality of performance records corresponding respectively to the test cases, wherein each performance record comprises differences in the performance parameters of the SUT from before and after the execution of a corresponding test case;
• identifying an outlier performance record from the second difference vector by comparing the resource usage metrics from first difference vector and the resource usage metrics from the second difference vector, wherein the outlier performance record represents a test case that has different resource usage metrics on the SUT and the baseline system;
• and
• determining a root cause of the soft failure of the SUT by executing an additional test case particular to the resource usage metrics and analyzing the additional test case corresponding in response to identifying the outlier performance record.

A computer program product comprising a computer-readable storage media having computer-executable instructions stored thereupon, which when executed by a processor cause the processor to perform a method for testing a system under test (SUT) in an active environment, the method comprising:

• generating a plurality of test cases for testing the SUT, the plurality of test cases generated based on a coverage model of the SUT, wherein the coverage model comprises a plurality of attributes;
• recording a first difference vector by executing the plurality of test cases on a baseline system, which is different from the SUT, and monitoring performance parameters of the baseline system before and after executing the plurality of test cases, wherein the first difference vector comprises a plurality of performance records corresponding respectively to the test cases, wherein each performance record comprises differences in the performance parameters of the baseline system from before and after the execution of a corresponding test case, the performance parameters comprising resource usage metrics of each of the plurality of test cases;
• recording a second difference vector by executing the plurality of test cases on the SUT and monitoring performance parameters of the SUT before and after executing the plurality of test cases, wherein the second difference vector also comprises a plurality of performance records corresponding respectively to the test cases, wherein each performance record comprises differences in the performance parameters of the SUT from before and after the execution of a corresponding test case;
• identifying an outlier performance record from the second difference vector by comparing the resource usage metrics from first difference vector and the resource usage metrics from the second difference vector, wherein the outlier performance record represents a test case that has different resource usage metrics on the SUT and the baseline system;
• and
• determining a root cause of the soft failure of the SUT by executing an additional test case particular to the resource usage metrics and analyzing the additional test case corresponding in response to identifying the outlier performance record.