# US 11,113,167 B1 · System testing infrastructure with hidden variable, hidden attribute, and hidden value detection > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,113,167 B1 - Document ID: US11113167B1 - Status: Granted - Kind: B1 - Application number: 17/121,825 - Filed: 2020-12-15 - Granted: 2021-09-07 - Earliest claimed: 2020-12-15 - Assignees: International Business Machines Corporation - Applicants: International Business Machines Corporation - Inventors: Andrew C. M. Hicks; Daniel Nicolas Gisolfi; Dale E. Blue; Ryan Thomas Rawlins - Web page: https://patentgrep.com/patent/US11113167B1 ## CPC - G06F 11/263: Generation of test inputs, e.g. test vectors, patterns or sequences {; with adaptation of the tested hardware for testability with external testers} - G06F 11/267: Reconfiguring circuits for testing, e.g. LSSD, partitioning - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract Inputs to a system under test (SUT) are modeled as a collection of attribute-value pairs. A set of testcases is executed using a set of test vectors that provides complete n-wise coverage of the attribute-value pairs. For each execution of the testcases, updating, for each execution of the set of testcases, for each testcase, a non-binary success rate (ST) based on the binary execution results. In response to a first success rate corresponding to a particular testcase being below a predetermined threshold, a second set of testcases is generated based on the test vectors. For each testcase, a second success rate (ST′) is computed based on a second set of execution results of a second set of testcases. In response to the second success rate corresponding to the particular testcase being substantially same as the first success rate, a user is notified of a defect in modeling the SUT inputs. ## Claims ### Claim 1 A method for detecting and localizing a fault when testing a system under test (SUT), the method comprising: modeling inputs to the SUT as a collection of attribute-value pairs; generating an initial set of test vectors that provides complete n-wise coverage of a test space represented by the attribute-value pairs; generating a set of testcases from the initial set of test vectors using combinatorics test design; executing the set of testcases to obtain a set of execution results, the execution results being in binary form indicative that a testcase succeeded or failed, the set of testcases executed a plurality of times; updating, for each execution of the set of testcases, for each testcase, a non-binary success rate (ST) based on the set of execution results; in response to a first success rate corresponding to a particular testcase being below a predetermined threshold: generating a second set of testcases based on the test vectors, the second set of testcases generated using inverse combinatorics test design; executing the second set of testcases to obtain a second set of execution results using a second set of test vectors, the second set of testcases executed at least a predetermined number of times; and updating, for each execution of the second set of testcases, for each testcase, a second success rate (ST′) based on the second set of execution results; and in response to the second success rate corresponding to the particular testcase being within a predetermined threshold of the first success rate, notifying a user of a defect in the modeling of the inputs to the SUT. ### Claim 2 (depends on 1) The method of claim 1, wherein the non-binary success rate ST is computed using Pseudo-Boolean algebra. ### Claim 3 (depends on 1) The method of claim 1, further comprising, in response to the second success rate corresponding to the particular testcase not being within the predetermined threshold of the first success rate, and in response to all of the testcases in the second set of testcases being successful, notifying the user that an attribute is to be added to test the SUT. ### Claim 4 (depends on 3) The method of claim 3, wherein in response to all of the testcases in the second set of testcases not being successful, outputting, to the user, an identification of the particular testcase to be used for diagnosing a soft failure associated with the SUT. ### Claim 5 (depends on 4) The method of claim 4, wherein a soft failure is caused by one or more operating conditions of the SUT. ### Claim 6 (depends on 1) The method of claim 1, wherein a Cartesian product space comprising all possible combinations of the attribute-value pairs is reduced to a reduced test space at least in part using combinatorics test designing. ### Claim 7 (depends on 6) The method of claim 6, wherein the Cartesian product space is further reduced based on an architectural restriction of the SUT. ### Claim 8 A system comprising: a memory; and a processor coupled with the memory, the processor configured to perform a method for detecting and localizing a fault when testing a system under test (SUT), the method comprising: modeling inputs to the SUT as a collection of attribute-value pairs; generating an initial set of test vectors that provides complete n-wise coverage of a test space represented by the attribute-value pairs; generating a set of testcases from the initial set of test vectors using combinatorics test design; executing the set of testcases to obtain a set of execution results, the execution results being in binary form indicative that a testcase succeeded or failed, the set of testcases executed a plurality of times; updating, for each execution of the set of testcases, for each testcase, a non-binary success rate (ST) based on the set of execution results; in response to a first success rate corresponding to a particular testcase being below a predetermined threshold: generating a second set of testcases based on the test vectors, the second set of testcases generated using inverse combinatorics test design; executing the second set of testcases to obtain a second set of execution results using a second set of test vectors, the second set of testcases executed at least a predetermined number of times; and updating, for each execution of the second set of testcases, for each testcase, a second success rate (ST′) based on the second set of execution results; and in response to the second success rate corresponding to the particular testcase being within a predetermined threshold of the first success rate, notifying a user of a defect in the modeling of the inputs to the SUT. ### Claim 9 (depends on 8) The system of claim 8, wherein the non-binary success rate ST is computed using Pseudo-Boolean algebra. ### Claim 10 (depends on 8) The system of claim 8, wherein the method further comprises, in response to the second success rate corresponding to the particular testcase not being within the predetermined threshold of the first success rate, and in response to all of the testcases in the second set of testcases being successful, notifying the user that an attribute is to be added to test the SUT. ### Claim 11 (depends on 10) The system of claim 10, wherein in response to all of the testcases in the second set of testcases not being successful, outputting, to the user, an identification of the particular testcase to be used for diagnosing a soft failure associated with the SUT. ### Claim 12 (depends on 11) The system of claim 11, wherein a soft failure is caused by one or more operating conditions of the SUT. ### Claim 13 (depends on 8) The system of claim 8, wherein a Cartesian product space comprising all possible combinations of the attribute-value pairs is reduced to a reduced test space at least in part using combinatorics test designing. ### Claim 14 (depends on 13) The system of claim 13, wherein the Cartesian product space is further reduced based on an architectural restriction of the SUT. ### Claim 15 A computer program product comprising one or more computer-readable storage media having computer-executable instructions stored thereupon, which when executed by a processor cause the processor to perform a method for detecting and localizing a fault when testing a system under test (SUT), the method comprising: modeling inputs to the SUT as a collection of attribute-value pairs; generating an initial set of test vectors that provides complete n-wise coverage of a test space represented by the attribute-value pairs; generating a set of testcases from the initial set of test vectors using combinatorics test design; executing the set of testcases to obtain a set of execution results, the execution results being in binary form indicative that a testcase succeeded or failed, the set of testcases executed a plurality of times; updating, for each execution of the set of testcases, for each testcase, a non-binary success rate (ST) based on the set of execution results; in response to a first success rate corresponding to a particular testcase being below a predetermined threshold: generating a second set of testcases based on the test vectors, the second set of testcases generated using inverse combinatorics test design; executing the second set of testcases to obtain a second set of execution results using a second set of test vectors, the second set of testcases executed at least a predetermined number of times; and updating, for each execution of the second set of testcases, for each testcase, a second success rate (ST) based on the second set of execution results; and in response to the second success rate corresponding to the particular testcase being within a predetermined threshold of the first success rate, notifying a user of a defect in the modeling of the inputs to the SUT. ### Claim 16 (depends on 15) The computer program product of claim 15, wherein the non-binary success rate ST is computed using Pseudo-Boolean algebra. ### Claim 17 (depends on 15) The computer program product of claim 15, wherein the method further comprises, in response to the second success rate corresponding to the particular testcase not being within the predetermined threshold of the first success rate, and in response to all of the testcases in the second set of testcases being successful, notifying the user that an attribute is to be added to test the SUT. ### Claim 18 (depends on 17) The computer program product of claim 17, wherein in response to all of the testcases in the second set of testcases not being successful, outputting, to the user, an identification of the particular testcase to be used for diagnosing a soft failure associated with the SUT. ### Claim 19 (depends on 15) The computer program product of claim 15, wherein a soft failure is caused by one or more operating conditions of the SUT. ### Claim 20 (depends on 15) The computer program product of claim 15, wherein a Cartesian product space comprising all possible combinations of the attribute-value pairs is reduced to a reduced test space at least in part using combinatorics test designing.