System testing infrastructure with hidden variable, hidden attribute, and hidden value detection

A system comprising:

• a memory;
• and
• a processor coupled with the memory, the processor configured to perform a method for detecting and localizing a fault when testing a system under test (SUT), the method comprising:
• modeling inputs to the SUT as a collection of attribute-value pairs;
• generating an initial set of test vectors that provides complete n-wise coverage of a test space represented by the attribute-value pairs;
• generating a set of testcases from the initial set of test vectors using combinatorics test design;
• executing the set of testcases to obtain a set of execution results, the execution results being in binary form indicative that a testcase succeeded or failed, the set of testcases executed a plurality of times;
• updating, for each execution of the set of testcases, for each testcase, a non-binary success rate (ST) based on the set of execution results;
• in response to a first success rate corresponding to a particular testcase being below a predetermined threshold:
• generating a second set of testcases based on the test vectors, the second set of testcases generated using inverse combinatorics test design;
• executing the second set of testcases to obtain a second set of execution results using a second set of test vectors, the second set of testcases executed at least a predetermined number of times;
• and
• updating, for each execution of the second set of testcases, for each testcase, a second success rate (ST′) based on the second set of execution results;
• and
• in response to the second success rate corresponding to the particular testcase being within a predetermined threshold of the first success rate, notifying a user of a defect in the modeling of the inputs to the SUT.

A computer program product comprising one or more computer-readable storage media having computer-executable instructions stored thereupon, which when executed by a processor cause the processor to perform a method for detecting and localizing a fault when testing a system under test (SUT), the method comprising:

• modeling inputs to the SUT as a collection of attribute-value pairs;
• generating an initial set of test vectors that provides complete n-wise coverage of a test space represented by the attribute-value pairs;
• generating a set of testcases from the initial set of test vectors using combinatorics test design;
• executing the set of testcases to obtain a set of execution results, the execution results being in binary form indicative that a testcase succeeded or failed, the set of testcases executed a plurality of times;
• updating, for each execution of the set of testcases, for each testcase, a non-binary success rate (ST) based on the set of execution results;
• in response to a first success rate corresponding to a particular testcase being below a predetermined threshold:
• generating a second set of testcases based on the test vectors, the second set of testcases generated using inverse combinatorics test design;
• executing the second set of testcases to obtain a second set of execution results using a second set of test vectors, the second set of testcases executed at least a predetermined number of times;
• and
• updating, for each execution of the second set of testcases, for each testcase, a second success rate (ST) based on the second set of execution results;
• and
• in response to the second success rate corresponding to the particular testcase being within a predetermined threshold of the first success rate, notifying a user of a defect in the modeling of the inputs to the SUT.