Identifying false positives in test case failures using combinatorics

A system comprising:

• a memory device;
• and
• one or more processors coupled with the memory device, the one or more processors configured to perform a method for detecting and localizing a fault in a system under test (SUT), the method comprising:
• modeling inputs to the SUT as a collection of attribute-value pairs;
• reducing a Cartesian product space comprising all possible combinations of the attribute-value pairs to a reduced test space;
• generating an initial set of test vectors that provides complete n-wise coverage of the reduced test space;
• generating an initial set of test cases from the initial set of test vectors;
• executing the initial set of test cases to obtain a first set of execution results;
• determining, based at least in part on the first set of execution results, a set of failing test cases;
• determining, based on a machine learning model, a subset of false positives from the set of failing test cases;
• generating a set of new test cases from a selected failing test case that is not in the subset of false positives, wherein generating the set of new test cases comprises generating, in relation to each attribute in the selected failing test case, a respective subset of new test cases at least in part by changing a respective attribute value for the attribute in the selected failing test case to each other candidate attribute value for the attribute that is not present in any of the one or more test cases that failed execution;
• executing the set of new test cases to obtain a second set of execution results;
• and
• detecting and localizing the fault based at least in part on the second set of execution results.

A computer program product comprising a memory storage device having computer executable instructions stored thereon, the computer executable instructions when executed by one or more processing units cause the one of more processing units to perform a method comprising:

• modeling inputs to the SUT as a collection of attribute-value pairs;
• reducing a Cartesian product space comprising all possible combinations of the attribute-value pairs to a reduced test space;
• generating an initial set of test vectors that provides complete n-wise coverage of the reduced test space;
• generating an initial set of test cases from the initial set of test vectors;
• executing the initial set of test cases to obtain a first set of execution results;
• determining, based at least in part on the first set of execution results, a set of failing test cases;
• determining, based on a machine learning model, a subset of false positives from the set of failing test cases;
• generating a set of new test cases from a selected failing test case that is not in the subset of false positives, wherein generating the set of new test cases comprises generating, in relation to each attribute in the selected failing test case, a respective subset of new test cases at least in part by changing a respective attribute value for the attribute in the selected failing test case to each other candidate attribute value for the attribute that is not present in any of the one or more test cases that failed execution;
• executing the set of new test cases to obtain a second set of execution results;
• and
• detecting and localizing the fault based at least in part on the second set of execution results.