# US 2024/0256432 A1 · Testing a machine learning model

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2024/0256432 A1
- Document ID: US20240256432A1
- Status: Published application
- Kind: A1
- Application number: 18/187,069
- Filed: 2023-03-21
- Published: 2024-08-01
- Earliest claimed: 2023-01-26
- Assignees: International Business Machines Corporation
- Applicants: International Business Machines Corporation
- Inventors: Daniel Jakub Ryszka; Lukasz G. Cmielowski; Jan Soltysik; Szymon Kucharczyk
- Web page: https://patentgrep.com/patent/US20240256432A1

## CPC

- G06F 11/3684: {for test design, e.g. generating new test cases}
- G06F 11/3676: {for coverage analysis}
- G06F 11/3688: {for test execution, e.g. scheduling of test suites}
- G06N 20/00: Machine learning
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

A method for generating tests of a machine learning (ML) engine includes determining a feature space of the set of configuration parameters of at least part of initial test cases of the ML engine. One or more areas of low coverage may be identified in the feature space. A configuration of a new test case may be defined such that the new test case covers a low coverage area of the feature space. The new test case may be provided for testing a performance of the ML engine to find a ML model using an input dataset according to the configuration of the new test case.

## Claims

### Claim 1

A computer-implemented method, the method comprising:
generating tests of a machine learning (ML) engine configured for automatically finding a ML Model for performing a machine learning task based on an input dataset, wherein initial test cases are provided for testing the ML engine and each test case has a specific configuration for testing the ML engine regarding performing a specific machine learning task, wherein each configuration is defined by a set of configuration parameters comprising at least part of a set of characteristics of the input dataset and/or at least part of a set of machine learning parameters, the tests generated by:
determining a feature space of the set of configuration parameters of at least part of the initial test cases, the feature space having a dimension equal to the number of configuration parameters;
identifying one or more areas of low coverage in the feature space;
defining a configuration of a new test case such that the new test case covers a low coverage area of the feature space; and
providing the new test case for testing a performance of the ML engine to find a ML model using an input dataset according to the configuration of the new test case.

### Claim 2 (depends on 1)

The computer-implemented method of claim 1, the at least part of the test cases comprising any one of:
all the provided tests cases; or
selected one or more test cases of the provided test cases.

### Claim 3 (depends on 1)

The computer-implemented method of claim 1, the method further comprising:
providing a bug tracking system that is configured to identify bugs when testing the ML engine using the initial test cases and collect bug data descriptive of the bugs; and
processing the bug data for identifying one or more test cases, referred to as bug test cases, wherein the at least part of the initial test cases used to determine the feature space are the bug test cases.

### Claim 4 (depends on 3)

The computer-implemented method of claim 3, wherein:
the bug data further indicating values of a problematic characteristic and/or a problematic machine learning parameter; and
the configuration of the new test case is defined further using the problematic characteristic and/or the problematic machine learning parameter.

### Claim 5 (depends on 3)

The computer-implemented method of claim 3, wherein the bug data of each test case indicating at least one of the group comprises:
an erroneous value of a performance metric for the performed machine learning task of the test case, wherein the performance metric comprises: an accuracy for a classification task or a mean-squared error for a regression task;
an exceeded processing time limit of the testing of the ML engine using the test case; or
different results using multiple runs of the same input dataset of the test case.

### Claim 6 (depends on 1)

The computer-implemented method of claim 1, further comprising defining further new test cases such that the feature space has no low coverage areas.

### Claim 7 (depends on 1)

The computer-implemented method of claim 1, the dataset comprising records, wherein each record of the records comprises values of features, the characteristics of the dataset comprise at least one of:
the fraction of categorical features in the dataset,
the number of features in the dataset,
the number of records in the dataset, or
the proportion of records with at least one missing feature.

### Claim 8 (depends on 1)

The computer-implemented method of claim 1, the set of configuration parameters comprising a subset of characteristics of the input datasets of the at least part of test cases, wherein the feature space is defined as a correlation matrix of the subset of characteristics.

### Claim 9 (depends on 8)

The computer-implemented method of claim 8, the feature space being a 2-dimensional space that is defined as a correlation matrix of the number of categorical features and the number of numerical features in each dataset of the at least part of test cases.

### Claim 10 (depends on 1)

The computer-implemented method of claim 1, wherein defining the test case comprises: using the defined configuration and a code of one of the provided test cases to generate a code for the new test case.

### Claim 11 (depends on 1)

The computer-implemented method of claim 1, wherein the testing of the performance of the ML engine comprises computing a performance index of the new test case, wherein the new test case is provided in case the performance index fulfils a predefined performance condition, the method further comprising saving the new test case with the initial test cases in case the performance index fulfils the predefined performance condition.

### Claim 12 (depends on 11)

The computer-implemented method of claim 11, wherein the performance index is determined using at least one of: execution time of the new test case, features coverage of the feature space including the new test case, number of found bugs with test cases of the feature space, or number of missed bugs with test cases of the feature space.

### Claim 13 (depends on 1)

The computer-implemented method of claim 1, wherein determining the feature space comprises:
projecting all data to a k-dimensional feature space, where k is the number of configuration parameters;
dividing the space into a number of regions;
constructing a histogram, where each region is represented by a histogram bar with height equal to number of datapoints in normalized feature space in this region; and
calculating a standard deviation, to detect if there's a spread in test coverage, wherein a first quartile of histogram bars is qualified as regions where coverage is low.

### Claim 14

A system comprising:
a processor; and
a memory in communication with the processor, the memory containing instructions that, when executed by the processor, cause the processor to:
generate tests of a machine learning (ML) engine configured for automatically finding a ML Model for performing a machine learning task based on an input dataset, wherein initial test cases are provided for testing the ML engine and each test case has a specific configuration for testing the ML engine regarding performing a specific machine learning task, wherein each configuration is defined by a set of configuration parameters comprising at least part of a set of characteristics of the input dataset and/or at least part of a set of machine learning parameters, the tests generated by:
determine a feature space of the set of configuration parameters of at least part of the initial test cases, the feature space having a dimension equal to the number of configuration parameters;
identify one or more areas of low coverage in the feature space;
define a configuration of a new test case such that the new test case covers a low coverage area of the feature space; and
provide the new test case for testing a performance of the ML engine to find a ML model using an input dataset according to the configuration of the new test case.

### Claim 15 (depends on 14)

The system of claim 14, the at least part of the test cases comprising any one of:
all the provided tests cases; or
selected one or more test cases of the provided test cases.

### Claim 16 (depends on 14)

The system of claim 14, the memory containing additional instructions that, when executed by the processor, cause the processor to:
provide a bug tracking system that is configured to identify bugs when testing the ML engine using the initial test cases and collect bug data descriptive of the bugs; and
process the bug data for identifying one or more test cases, referred to as bug test cases, wherein the at least part of the initial test cases used to determine the feature space are the bug test cases.

### Claim 17 (depends on 14)

The system of claim 14, wherein:
the bug data further indicating values of a problematic characteristic and/or a problematic machine learning parameter; and
the configuration of the new test case is defined further using the problematic characteristic and/or the problematic machine learning parameter.

### Claim 18 (depends on 17)

The system of claim 17, wherein the bug data of each test case indicating at least one of the group comprises:
an erroneous value of a performance metric for the performed machine learning task of the test case, wherein the performance metric comprises: an accuracy for a classification task or a mean-squared error for a regression task;
an exceeded processing time limit of the testing of the ML engine using the test case; or
different results using multiple runs of the same input dataset of the test case.

### Claim 19 (depends on 14)

The system of claim 14, the memory containing additional instructions that, when executed by the processor, cause the processor to define further new test cases such that the feature space has no low coverage areas.

### Claim 20

A computer program product comprising a computer-readable storage medium having computer-readable program code embodied therewith, the computer-readable program code configured to cause a computer to:
generate tests of a machine learning (ML) engine configured for automatically finding a ML Model for performing a machine learning task based on an input dataset, wherein initial test cases are provided for testing the ML engine and each test case has a specific configuration for testing the ML engine regarding performing a specific machine learning task, wherein each configuration is defined by a set of configuration parameters comprising at least part of a set of characteristics of the input dataset and/or at least part of a set of machine learning parameters, the tests generated by:
determine a feature space of the set of configuration parameters of at least part of the initial test cases, the feature space having a dimension equal to the number of configuration parameters;
identify one or more areas of low coverage in the feature space;
define a configuration of a new test case such that the new test case covers a low coverage area of the feature space; and
provide the new test case for testing a performance of the ML engine to find a ML model using an input dataset according to the configuration of the new test case.
