Automated machine learning test system

A computing device comprising:

• a processor;
• and
• a computer-readable medium operably coupled to the processor, the computer-readable medium having computer-readable instructions stored thereon that, when executed by the processor, cause the computing device to
• execute software under test with a first plurality of test configurations to generate a test result for each test configuration of the first plurality of test configurations, wherein each test configuration of the first plurality of test configurations includes a value for each test parameter of the plurality of test parameters, wherein each test parameter of the plurality of test parameters is an input to the software under test, wherein the test result includes at least one of a severity code, a memory consumption value, or an execution time value, wherein the severity code indicates whether or not execution of the software under test failed using a respective test configuration, wherein the memory consumption value indicates an amount of computer memory used for execution of the software under test using the respective test configuration, wherein the execution time value indicates an amount of time used for execution of the software under test using the respective test configuration;
• train a predictive model using each test configuration of the first plurality of test configurations and a target variable value that is the test result generated for each test configuration of the first plurality of test configurations based on an objective function value;
• execute the trained predictive model with a second plurality of test configurations to predict the test result for each test configuration of the second plurality of test configurations, wherein each test configuration of the second plurality of test configurations includes the value for each test parameter of the plurality of test parameters;
• select a predefined number of test configurations from the second plurality of test configurations based on the predicted test result for each test configuration of the second plurality of test configurations, wherein the selected predefined number of test configurations define a third plurality of test configurations, wherein the third plurality of test configurations are selected to maximize the memory consumption value or the execution time value or to result in failed execution of the software under test;
• execute the software under test with the defined third plurality of test configurations to generate the test result for each test configuration of the third plurality of test configurations, wherein each test configuration of the third plurality of test configurations includes the value for each test parameter of the plurality of test parameters;
• (AA) retrain the predictive model using each test configuration of the first plurality of test configurations in association with the test result generated for each test configuration of the first plurality of test configurations and using each test configuration of the third plurality of test configurations in association with the test result generated for each test configuration of the third plurality of test configurations;
• (BB) update the second plurality of test configurations by removing the third plurality of test configurations from the second plurality of test configurations;
• (CC) execute the predictive model retrained in (AA) with the updated second plurality of test configurations to predict the test result for each test configuration of the second plurality of test configurations;
• (DD) select the predefined number of test configurations from the updated second plurality of test configurations based on the predicted test result for each test configuration of the second plurality of test configurations in (CC), wherein the selected predefined number of test configurations define a fourth plurality of test configurations, wherein each test configuration of the fourth plurality of test configurations includes the value for each test parameter of the plurality of test parameters, wherein each test configuration of the fourth plurality of test configurations is unique relative to other test configurations of the fourth plurality of test configurations and is unique relative to the first plurality of test configurations and relative to the third plurality of test configurations;
• (EE) execute the software under test with the fourth plurality of test configurations to generate the test result for each test configuration of the fourth plurality of test configurations;
• and
• output the generated test result for each test configuration of the first plurality of test configurations, for each test configuration of the third plurality of test configurations, and for each test configuration of the fourth plurality of test configurations to identify errors in the software under test.

A method of automatically defining new test configurations for testing software, the method comprising:

• executing, by a computing device, software under test with a first plurality of test configurations to generate a test result for each test configuration of the first plurality of test configurations, wherein each test configuration of the first plurality of test configurations includes a value for each test parameter of the plurality of test parameters, wherein each test parameter of the plurality of test parameters is an input to the software under test, wherein the test result includes at least one of a severity code, a memory consumption value, or an execution time value, wherein the severity code indicates whether or not execution of the software under test failed using a respective test configuration, wherein the memory consumption value indicates an amount of computer memory used for execution of the software under test using the respective test configuration, wherein the execution time value indicates an amount of time used for execution of the software under test using the respective test configuration;
• training, by the computing device, a predictive model using each test configuration of the first plurality of test configurations and a target variable value that is the test result generated for each test configuration of the first plurality of test configurations based on an objective function value;
• executing, by the computing device, the trained predictive model with a second plurality of test configurations to predict the test result for each test configuration of the second plurality of test configurations, wherein each test configuration of the second plurality of test configurations includes the value for each test parameter of the plurality of test parameters;
• selecting, by the computing device, a predefined number of test configurations from the second plurality of test configurations based on the predicted test result for each test configuration of the second plurality of test configurations, wherein the selected predefined number of test configurations define a third plurality of test configurations, wherein the third plurality of test configurations are selected to maximize the memory consumption value or the execution time value or to result in failed execution of the software under test;
• executing, by the computing device, the software under test with the defined third plurality of test configurations to generate the test result for each test configuration of the third plurality of test configurations, wherein each test configuration of the third plurality of test configurations includes the value for each test parameter of the plurality of test parameters;
• (AA) retraining, by the computing device, the predictive model using each test configuration of the first plurality of test configurations in association with the test result generated for each test configuration of the first plurality of test configurations and using each test configuration of the third plurality of test configurations in association with the test result generated for each test configuration of the third plurality of test configurations;
• (BB) updating, by the computing device, the second plurality of test configurations by removing the third plurality of test configurations from the second plurality of test configurations;
• (CC) executing, by the computing device, the predictive model trained in (AA) with the updated second plurality of test configurations to predict the test result for each test configuration of the second plurality of test configurations;
• (DD) selecting, by the computing device, the predefined number of test configurations from the updated second plurality of test configurations based on the predicted test result for each test configuration of the second plurality of test configurations in (CC), wherein the selected predefined number of test configurations define a fourth plurality of test configurations, wherein each test configuration of the fourth plurality of test configurations includes the value for each test parameter of the plurality of test parameters, wherein each test configuration of the fourth plurality of test configurations is unique relative to other test configurations of the fourth plurality of test configurations and is unique relative to the first plurality of test configurations and relative to the third plurality of test configurations;
• (EE) executing, by the computing device, the software under test with the fourth plurality of test configurations to generate the test result for each test configuration of the fourth plurality of test configurations;
• and
• outputting, by the computing device, the generated test result for each test configuration of the first plurality of test configurations, for each test configuration of the third plurality of test configurations, and for each test configuration of the fourth plurality of test configurations to identify errors in the software under test.