System testing infrastructure with hidden variable, hidden attribute, and hidden value detection

A system comprising:

• a memory;
• and
• a processor coupled with the memory, the processor is configured to perform a method for detecting and localizing a fault when testing a system under test (SUT), the method comprising:
• modeling inputs to the SUT as a collection of attribute-value pairs;
• generating an initial set of test vectors that provides complete n-wise coverage of a test space represented by the attribute-value pairs;
• generating a set of testcases from the initial set of test vectors;
• executing the set of testcases to obtain a set of execution results, the execution results being in binary form indicative that a testcase succeeded or failed, the set of testcases executed a plurality of times;
• updating, for each execution of the set of testcases, for each attribute-value pair, a non-binary success rate (SAV) based on the execution results, the non-binary success rate of an attribute-value pair is based on execution results of each testcase that uses said attribute-value pair;
• and
• outputting, to a user, in response to the success rate SAV of the attribute-value pair being below a predetermined threshold, an identification of one or more testcases that use the attribute-value pair, wherein the one or more testcases are to be used for diagnosing a soft failure associated with the SUT.

A computer program product comprising a computer-readable storage media having computer-executable instructions stored thereupon, which when executed by a processor cause the processor to perform for detecting and localizing a fault when testing a system under test (SUT), the method comprising:

• modeling inputs to the SUT as a collection of attribute-value pairs;
• generating an initial set of test vectors that provides complete n-wise coverage of a test space represented by the attribute-value pairs;
• generating a set of testcases from the initial set of test vectors;
• executing the set of testcases to obtain a set of execution results, the execution results being in binary form indicative that a testcase succeeded or failed, the set of testcases executed a plurality of times;
• updating, for each execution of the set of testcases, for each attribute-value pair, a non-binary success rate (SAV) based on the execution results, the non-binary success rate of an attribute-value pair is based on execution results of each testcase that uses said attribute-value pair;
• and
• outputting, to a user, in response to the success rate SAV of the attribute-value pair being below a predetermined threshold, an identification of one or more testcases that use the attribute-value pair, wherein the one or more testcases are to be used for diagnosing a soft failure associated with the SUT.