Testing a machine learning model

A system comprising:

• a processor;
• and
• a memory in communication with the processor, the memory containing instructions that, when executed by the processor, cause the processor to:
• generate tests of a machine learning (ML) engine configured for automatically finding a ML Model for performing a machine learning task based on an input dataset, wherein initial test cases are provided for testing the ML engine and each test case has a specific configuration for testing the ML engine regarding performing a specific machine learning task, wherein each configuration is defined by a set of configuration parameters comprising at least part of a set of characteristics of the input dataset and/or at least part of a set of machine learning parameters, the tests generated by:
• determine a feature space of the set of configuration parameters of at least part of the initial test cases, the feature space having a dimension equal to a number of configuration parameters, wherein the feature space is based on bug cases;
• identify one or more areas of low coverage in the feature space;
• define a configuration of a new test case such that the new test case covers a low coverage area of the feature space;
• and
• provide the new test case for testing a performance of the ML engine to find a ML model using an input dataset according to the configuration of the new test case.

A computer program product comprising a computer-readable storage medium having computer-readable program code embodied therewith, the computer-readable program code configured to cause a computer to:

• generate tests of a machine learning (ML) engine configured for automatically finding a ML Model for performing a machine learning task based on an input dataset, wherein initial test cases are provided for testing the ML engine and each test case has a specific configuration for testing the ML engine regarding performing a specific machine learning task, wherein each configuration is defined by a set of configuration parameters comprising at least part of a set of characteristics of the input dataset and/or at least part of a set of machine learning parameters, the tests generated by:
• determine a feature space of the set of configuration parameters of at least part of the initial test cases, the feature space having a dimension equal to a number of configuration parameters, wherein determining the feature space comprises:
• projecting all data to a k-dimensional feature space, where k is the number of configuration parameters;
• dividing the space into a number of regions;
• constructing a histogram, where each region is represented by a histogram bar with height equal to number of datapoints in normalized feature space in this region;
• and
• calculating a standard deviation, to detect if there's a spread in test coverage, wherein a first quartile of histogram bars is qualified as regions where coverage is low;
• identify one or more areas of low coverage in the feature space;
• define a configuration of a new test case such that the new test case covers a low coverage area of the feature space;
• and
• provide the new test case for testing a performance of the ML engine to find a ML model using an input dataset according to the configuration of the new test case.