# US 11,442,839 B2 · Runtime metrics based test ordering > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,442,839 B2 - Document ID: US11442839B2 - Status: Granted - Kind: B2 - Application number: 16/813,792 - Filed: 2020-03-10 - Published: 2021-09-16 - Granted: 2022-09-13 - Earliest claimed: 2020-03-10 - Assignees: International Business Machines Corporation - Applicants: International Business Machines Corporation - Inventors: Andrew C. M. Hicks; Ryan Thomas Rawlins; Deborah A. Furman - Web page: https://patentgrep.com/patent/US11442839B2 ## CPC - G06F 11/3612: {by runtime analysis (performance monitoring G06F11/3466)} - G06F 11/3676: {for coverage analysis} - G06F 11/3684: {for test design, e.g. generating new test cases} - G06F 11/3688: {for test execution, e.g. scheduling of test suites} - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract Techniques for runtime metrics based test ordering in a computer system are described herein. An aspect includes determining a first runtime metric associated with a first module based on the execution of a first test case. Another aspect includes determining a second runtime metric associated with a second module based on the execution of a second test case. Another aspect includes comparing the first runtime metric and the second runtime metric. Another aspect includes determining an order of a plurality of test cases based on the comparison of the first runtime metric and the second runtime metric. Another aspect includes executing the plurality of test cases in the determined order. ## Claims ### Claim 1 A computer-implemented method comprising: determining, by a processor, a bucket comprising an ordered plurality of test cases corresponding to a testing run of a system under test (SUT), the ordered plurality of test cases comprising a first test case and a second test case; and starting the testing run, wherein starting the testing run comprises: executing, by the processor, the first test case against the SUT, wherein executing the first test case comprises executing a first module; determining a first end state and a first runtime metric associated with the first module based on the execution of the first test case; executing the second test case against the SUT, wherein executing the second test case comprises executing a second module; determining a second end state and a second runtime metric associated with the second module based on the execution of the second test case; determining that the first end state and the second end state are the same, the first end state including a first fingerprint that identifies a first code path covered by the first test case, and the second end state including a second fingerprint that identifies a second code path covered by the second test case; and based on determining that the first end state and the second end state are the same: comparing the first runtime metric comprising a first memory utilization and the second runtime metric comprising a second memory utilization to determine that the first memory utilization is less than the second memory utilization; reordering the plurality of test cases in the bucket based on the comparison of the first runtime metric having the first memory utilization and the second runtime metric having the second memory utilization; continuing the testing run based on the reordered plurality of test cases in the bucket; during a subsequent iteration, comparing the first runtime metric comprising a first I/O transactions and the second runtime metric comprising a second I/O transactions to determine that the first I/O transactions are less than the second I/O transactions; and selecting the first test case for execution before the second test case based at least in part on the first I/O transactions being less than the second I/O transactions. ### Claim 2 (depends on 1) The method of claim 1, further comprising storing the first runtime metric, the second runtime metric, the first end state, and the second end state in a runtime data repository. ### Claim 3 (depends on 1) The method of claim 1, wherein reordering the plurality of test cases in the bucket further comprises reordering the plurality of test cases to prioritize the first module over the second module. ### Claim 4 (depends on 1) The method of claim 1, wherein the first and second runtime metrics comprise one of elapsed execution time, memory utilization, central processing unit (CPU) time, and a number of Input/Output (I/O) transactions. ### Claim 5 (depends on 1) The method of claim 1, wherein the plurality of cases comprises a third test case that includes the second module, and further comprising not executing the third test case of the plurality of test cases during the testing run corresponding to the bucket based on the comparison of the first runtime metric and the second runtime metric. ### Claim 6 (depends on 1) The method of claim 1, further comprising storing the first runtime metric, the first fingerprint, and a reference to where a copy of the first test case is located, the storing in a runtime data repository. ### Claim 7 (depends on 1) The method of claim 1, wherein the reordering comprises recalculating a binary decision diagram corresponding to the testing run. ### Claim 8 A system comprising: a memory having computer readable instructions; and one or more processors for executing the computer readable instructions, the computer readable instructions controlling the one or more processors to perform operations comprising: determining a bucket comprising an ordered plurality of test cases corresponding to a testing run of a system under test (SUT), the ordered plurality of test cases comprising a first test case and a second test case; and starting the testing run, wherein starting the testing run comprises: executing, by the processor, the first test case against the SUT, wherein executing the first test case comprises executing a first module; determining a first end state and a first runtime metric associated with the first module based on the execution of the first test case; executing the second test case against the SUT, wherein executing the second test case comprises executing a second module; determining a second end state and a second runtime metric associated with the second module based on the execution of the second test case; determining that the first end state and the second end state are the same; and based on determining that the first end state and the second end state are the same: comparing the first runtime metric comprising a first memory utilization and the second runtime metric comprising a second memory utilization to determine that the first test case is more efficient than the second test case based at least in part on the first memory utilization being less than the second memory utilization; modifying the plurality of test cases in the bucket based on the comparison of the first runtime metric having the first memory utilization and the second runtime metric having the second memory utilization, the modifying comprising recalculating a binary decision diagram corresponding to the testing run; continuing the testing run based on the reordered plurality of test cases in the bucket; during a subsequent iteration, comparing the first runtime metric comprising a first I/O transactions and the second runtime metric comprising a second I/O transactions to determine that the first I/O transactions are less than the second I/O transactions; and selecting the first test case for execution before the second test case based at least in part on the first I/O transactions being less than the second I/O transactions. ### Claim 9 (depends on 8) The system of claim 8, further comprising storing the first runtime metric, the second runtime metric, the first end state, and the second end state in a runtime data repository. ### Claim 10 (depends on 8) The system of claim 8, wherein modifying the plurality of test cases in the bucket further comprises reordering the plurality of test cases to prioritize the first module over the second module. ### Claim 11 (depends on 8) The system of claim 8, wherein the first and second runtime metrics comprise one of elapsed execution time, memory utilization, central processing unit (CPU) time, and a number of Input/Output (I/O) transactions. ### Claim 12 (depends on 8) The system of claim 8, wherein the plurality of cases comprises a third test case that includes the second module, and further comprising not executing the third test case of the plurality of test cases during the testing run corresponding to the bucket based on the comparison of the first runtime metric and the second runtime metric. ### Claim 13 A computer program product comprising a computer readable storage medium having program instructions embodied therewith, the program instructions executable by one or more processors to cause the one or more processors to perform operations comprising: determining a bucket comprising an ordered plurality of test cases corresponding to a testing run of a system under test (SUT), the ordered plurality of test cases comprising a first test case and a second test case; and starting the testing run, wherein starting the testing run comprises: executing, by the processor, the first test case against the SUT, wherein executing the first test case comprises executing a first module; determining a first end state and a first runtime metric associated with the first module based on the execution of the first test case; executing the second test case against the SUT, wherein executing the second test case comprises executing a second module; determining a second end state and a second runtime metric associated with the second module based on the execution of the second test case; determining that the first end state and the second end state are the same, the first end state including a first fingerprint that identifies a first code path covered by the first test case, and the second end state including a second fingerprint that identifies a second code path covered by the second test case; and based on determining that the first end state and the second end state are the same: comparing the first runtime metric comprising a first memory utilization and the second runtime metric comprising a second memory utilization to determine that the first memory utilization is less than the second memory utilization; reordering the plurality of test cases in the bucket based on the comparison of the first runtime metric having the first memory utilization and the second runtime metric having the second memory utilization, the reordering comprising recalculating a binary decision diagram corresponding to the testing run; continuing the testing run based on the reordered plurality of test cases in the bucket; during a subsequent iteration, comparing the first runtime metric comprising a first I/O transactions and the second runtime metric comprising a second I/O transactions to determine that the first I/O transactions are less than the second I/O transactions; and selecting the first test case for execution before the second test case based at least in part on the first I/O transactions being less than the second I/O transactions. ### Claim 14 (depends on 13) The computer program product of claim 13, further comprising storing the first runtime metric, the second runtime metric, the first end state, and the second end state in a runtime data repository. ### Claim 15 (depends on 13) The computer program product of claim 13, wherein reordering the plurality of test cases in the bucket further comprises reordering the plurality of test cases to prioritize the first module over the second module. ### Claim 16 (depends on 13) The computer program product of claim 13, wherein the first and second runtime metrics comprise one of elapsed execution time, memory utilization, central processing unit (CPU) time, and a number of Input/Output (I/O) transactions.