# US 10,482,002 B2 · Multi-layer test suite generation > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,482,002 B2 - Document ID: US10482002B2 - Status: Granted - Kind: B2 - Application number: 16/137,038 - Filed: 2018-09-20 - Published: 2019-01-24 - Granted: 2019-11-19 - Earliest claimed: 2016-08-24 - Assignees: Google Inc. - Applicants: Google LLC - Inventors: Keun Soo Yim; Iliyan Malchev - Web page: https://patentgrep.com/patent/US10482002B2 ## CPC - G06F 11/3684: {for test design, e.g. generating new test cases} - G06F 8/00: Arrangements for software engineering (testing or debugging G06F11/36; administrative, planning or organisation aspects of software project management G06Q10/06) - G06F 8/71: Version control (security arrangements therefor G06F21/57); Configuration management - G06F 9/44536: {Selecting among different versions} - G06F 11/36: Prevention of errors by analysis, debugging or testing of software - G06F 11/368: {for test version control, e.g. updating test cases to a new software version} - G06F 11/3692: {for test results analysis} - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract A system and method for developing a multi-layered test suite for an operating platform including a framework layer and a system layer includes executing a first test suite against a version of the operating platform modified based on a software faults. A first counter is incremented if a first test suite executed against the modified version of the operating platform fails. A second test suite can be executed against the modified version of the operating platform and test cases may be added to the first test suite based on whether the second test suite passes or fails. ## Claims ### Claim 1 A method for developing a test suite for an operating platform, the method comprising: performing a plurality of iterations, each of the plurality of iterations corresponding to a selected one of a plurality of software faults, each iteration including: executing a first test suite against a modified version of the operating platform corresponding to the selected one of a plurality of software faults, the first test suite including a first plurality of test cases; incrementing a first counter if executing the first test suite against the modified version of the operating platform results in at least one of the first plurality of test cases failing; executing a second test suite against the modified version of the operating platform, the second test suite including a second plurality of test cases; incrementing a second counter if executing the second test suite against the modified version of the operating platform results in at least one of the second plurality of test cases failing; incrementing a third counter if executing the second test suite against the modified version of the operating platform results in none of the second plurality of test cases failing; determining a ratio of the first counter to a sum of the second and third counters; performing another iteration of the plurality of iterations if the ratio fails to satisfy a first threshold; and ceasing iteration through the plurality of iterations if the ratio satisfies the first threshold. ### Claim 2 (depends on 1) The method of claim 1, wherein executing the second test suite against the modified version of the operating platform occurs if none of the first plurality of test cases fails when executing the first test suite against the modified version of the operating platform. ### Claim 3 (depends on 1) The method of claim 1 further comprising adding a test case to the first plurality of test cases when executing the second test suite against the modified version of the operating platform results in at least one of the second plurality of test cases failing. ### Claim 4 (depends on 3) The method of claim 3, wherein adding the test case to the first plurality of test cases occurs prior to performing the next iteration of the plurality of iterations. ### Claim 5 (depends on 1) The method of claim 1 further comprising flagging the selected one of the plurality of software faults if executing the second test suite against the modified version of the operating platform results in at least one of the second plurality of test cases failing. ### Claim 6 (depends on 5) The method of claim 5 further comprising developing a test case based on the flagged selected one of the plurality of software faults. ### Claim 7 (depends on 5) The method of claim 5 further comprising adding a test case to the first plurality of test cases based on the flagged selected one of the plurality of software faults. ### Claim 8 (depends on 1) The method of claim 1, wherein executing the first test suite against the modified version of the operating platform is performed if the first test suite includes at least one test case. ### Claim 9 (depends on 1) The method of claim 1, wherein performing the next iteration is further based on whether a quantity of previously performed iterations fails to satisfy a second threshold. ### Claim 10 (depends on 1) The method of claim 1, wherein the modified version of the operating platform has been created by altering a baseline version of the operating platform to include the selected one of the plurality of software faults. ### Claim 11 A system for developing a test suite for an operating platform, the system comprising: a processor; and a memory storing instructions that when executed by the processor, cause the processor to perform a plurality of iterative operations, each of the plurality of iterative operations including: executing a first test suite against a modified version of the operating platform, the modified version of the operating platform being created by altering a baseline version of the operating platform; incrementing a first counter if executing the first test suite against the modified version of the operating platform fails; executing a second test suite against the modified version of the operating platform if the first test suite against the modified version of the operating platform passes; incrementing a second counter when executing the second test suite against the modified version of the operating platform fails; incrementing a third counter if executing the second test suite against the modified version of the operating platform passes; determining a ratio of the first counter to a sum of the second and third counters, performing another of the plurality of iterative operations if the ratio fails to satisfy a first threshold; and ceasing performance of the plurality of iterative operations if the ratio satisfies the first threshold. ### Claim 12 (depends on 11) The system of claim 11, wherein each of the plurality of iterative operations further includes adding a test case to the first test suite if executing the second test suite against the modified version of the operating platform fails. ### Claim 13 (depends on 12) The system of claim 12, wherein adding the test case to the first test suite occurs prior to performing another of the plurality of iterative operations. ### Claim 14 (depends on 11) The system of claim 11, wherein executing the first test suite against the modified version of the operating platform is performed if the first test suite includes at least one test case. ### Claim 15 (depends on 11) The system of claim 11, wherein performing another of the plurality of iterative operations is further based on whether a quantity of previously performed iterative operations of the plurality of iterative operations fails to satisfy a second threshold. ### Claim 16 (depends on 11) The system of claim 11, wherein altering the baseline version of the operating platform is performed by injecting one of a plurality of software faults into the baseline version. ### Claim 17 (depends on 16) The system of claim 16, wherein each of the plurality of iterative operations corresponds to each of the plurality of software faults. ### Claim 18 A method for creating a test suite for an operating platform, the method comprising: performing a plurality of iterations corresponding to a plurality of software faults, each of the plurality of iterations corresponding to one of the plurality of software faults, and each of the plurality of iterations including: creating a modified version of the operating platform by altering a baseline version of the operating platform by injecting a corresponding one of the plurality of software faults into the baseline version of the operating platform; executing a first test suite against the modified version of the operating platform if the first test suite includes at least one test case; if executing the first test suite against the modified version of the operating platform fails, incrementing a first counter; and if executing the first test suite against the modified version of the operating platform passes: executing a second test suite against the modified version of the operating platform; if executing the second test suite against the modified version of the operating platform fails, incrementing a second counter; and adding a new test case to the first test suite for the corresponding one of the plurality of software faults; and if executing the second test suite against the modified version of the operating platform fails: if executing the second test suite against the modified version of the operating platform passes, incrementing a third counter; determining a ratio of the first counter to a sum of the second and third counters; if the ratio fails to satisfy a first threshold, performing another of the plurality of iterations; and if the ratio satisfies the first threshold, ceasing performance of the plurality of iterations. ### Claim 19 (depends on 18) The method of claim 18 wherein the performing another of the plurality of iterations is further based on whether a quantity of previously performed iterations of the plurality of iterations fails to satisfy a second threshold. ### Claim 20 (depends on 19) The method of claim 19 wherein the second threshold is based on a ratio of the previously performed iterations of the plurality of iterations to a total of the plurality of iterations.