Method for efficiently improving test case coverage and robustness

An information handling system comprising:

• a central processing unit (CPU);
• and
• a memory, accessible to the CPU, including processor executable instructions that, when executed by the CPU, cause the system to perform test case selection operations comprising:
• determining differences between first and second versions of firmware;
• identifying, based on the differences between the first and second versions, one or more impacted functions;
• reading by a customized loader implemented in a baseboard management controller (BMC), original binary code for the second version from a non-volatile memory of the BMC;
• loading binary code for the second version of firmware into a random access memory of a device under test;
• modifying the binary code for the second version of firmware in situ in the random access memory of the device under test to cause the one or more impacted functions, when called, to return an error value, wherein modifying the binary code comprises:
• determining, by the customized loader operating in conjunction with a configuration file indicating memory positions and corresponding values to modify, for each impacted function of the one or more impacted functions, configuration information indicative of:
• a return type (RTYPE) of the respective impacted function;
• and
• an instruction location (ILOC) for an instruction loading a return value to a return value register;
• determining, based on the RTYPE and a CPU architecture of the device under test, instruction data (IDATA) for loading the return value register with the error value;
• prior to overwriting the binary code at the ILOC, reading the binary code at the ILOC and storing the binary code for subsequently rolling back the binary code at the ILOC;
• and
• overwriting the binary code at the ILOC with the IDATA;
• running a plurality of test cases once on the modified binary code;
• identifying any test cases returning the error value as selected test cases;
• and
• including the selected test cases within target test cases for performing regression testing of the second version;
• accessing mapping data indicative of existing associations between functions and the selected test cases;
• and
• excluding, from the plurality of test cases run on the modified binary code, existing test cases, comprising any test case indicated by the mapping data as having an existing association with any of the one or more impacted functions.