Runtime metrics based test ordering

A system comprising:

• a memory having computer readable instructions;
• and
• one or more processors for executing the computer readable instructions, the computer readable instructions controlling the one or more processors to perform operations comprising:
• determining a bucket comprising an ordered plurality of test cases corresponding to a testing run of a system under test (SUT), the ordered plurality of test cases comprising a first test case and a second test case;
• and
• starting the testing run, wherein starting the testing run comprises:
• executing, by the processor, the first test case against the SUT, wherein executing the first test case comprises executing a first module;
• determining a first end state and a first runtime metric associated with the first module based on the execution of the first test case;
• executing the second test case against the SUT, wherein executing the second test case comprises executing a second module;
• determining a second end state and a second runtime metric associated with the second module based on the execution of the second test case;
• determining that the first end state and the second end state are the same;
• and
• based on determining that the first end state and the second end state are the same:
• comparing the first runtime metric comprising a first memory utilization and the second runtime metric comprising a second memory utilization to determine that the first test case is more efficient than the second test case based at least in part on the first memory utilization being less than the second memory utilization;
• modifying the plurality of test cases in the bucket based on the comparison of the first runtime metric having the first memory utilization and the second runtime metric having the second memory utilization, the modifying comprising recalculating a binary decision diagram corresponding to the testing run;
• continuing the testing run based on the reordered plurality of test cases in the bucket;
• during a subsequent iteration, comparing the first runtime metric comprising a first I/O transactions and the second runtime metric comprising a second I/O transactions to determine that the first I/O transactions are less than the second I/O transactions;
• and
• selecting the first test case for execution before the second test case based at least in part on the first I/O transactions being less than the second I/O transactions.

A computer program product comprising a computer readable storage medium having program instructions embodied therewith, the program instructions executable by one or more processors to cause the one or more processors to perform operations comprising:

• determining a bucket comprising an ordered plurality of test cases corresponding to a testing run of a system under test (SUT), the ordered plurality of test cases comprising a first test case and a second test case;
• and
• starting the testing run, wherein starting the testing run comprises:
• executing, by the processor, the first test case against the SUT, wherein executing the first test case comprises executing a first module;
• determining a first end state and a first runtime metric associated with the first module based on the execution of the first test case;
• executing the second test case against the SUT, wherein executing the second test case comprises executing a second module;
• determining a second end state and a second runtime metric associated with the second module based on the execution of the second test case;
• determining that the first end state and the second end state are the same, the first end state including a first fingerprint that identifies a first code path covered by the first test case, and the second end state including a second fingerprint that identifies a second code path covered by the second test case;
• and
• based on determining that the first end state and the second end state are the same:
• comparing the first runtime metric comprising a first memory utilization and the second runtime metric comprising a second memory utilization to determine that the first memory utilization is less than the second memory utilization;
• reordering the plurality of test cases in the bucket based on the comparison of the first runtime metric having the first memory utilization and the second runtime metric having the second memory utilization, the reordering comprising recalculating a binary decision diagram corresponding to the testing run;
• continuing the testing run based on the reordered plurality of test cases in the bucket;
• during a subsequent iteration, comparing the first runtime metric comprising a first I/O transactions and the second runtime metric comprising a second I/O transactions to determine that the first I/O transactions are less than the second I/O transactions;
• and
• selecting the first test case for execution before the second test case based at least in part on the first I/O transactions being less than the second I/O transactions.