# US 2023/0054582 A1 · Feature selection and hyperparameter optimization using lds

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2023/0054582 A1
- Document ID: US20230054582A1
- Status: Published application
- Kind: A1
- Application number: 17/408,650
- Filed: 2021-08-23
- Published: 2023-02-23
- Earliest claimed: 2021-08-23
- Assignees: International Business Machines Corporation
- Applicants: International Business Machines Corporation
- Inventors: Akihiro Kishimoto; Toshiyuki Hama
- Web page: https://patentgrep.com/patent/US20230054582A1

## CPC

- G06N 3/0985: Hyperparameter optimisation; Meta-learning; Learning-to-learn
- G06N 20/00: Machine learning
- G06N 7/08: using chaos models or non-linear system models
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

A computer-implemented method for simultaneous feature selection and hyperparameter optimization of non-linear models of machine learning is provided including setting a first solution having first hyperparameters and a first set of features of a plurality of features of a training data set, initializing a weight table providing a score for each feature of the first set of features, and initializing a discrepancy. The method further includes performing a limited discrepancy search (LDS), according to an order based on the weight table, to obtain a second solution having second hyperparameters and a second set of features by swapping the first set of features and switching the first hyperparameters from the first solution with the discrepancy, while updating the weight table during LDS, comparing the second solution with the first solution, and obtaining a new solution with improved features and hyperparameters, as an optimized solution.

## Claims

### Claim 1

A computer-implemented method for simultaneous feature selection and hyperparameter optimization of non-linear models of machine learning, the method comprising:
setting a first solution having first hyperparameters and a first set of features selected from a plurality of features of a training data set;
initializing a weight table providing a score for each feature of the first set of features;
initializing a discrepancy;
performing a limited discrepancy search (LDS), according to an order based on the weight table, to obtain a second solution having second hyperparameters and a second set of features from the plurality of features by swapping the first set of features and switching the first hyperparameters from the first solution with the discrepancy, while updating the weight table during LDS;
comparing the second solution with the first solution; and
obtaining a new solution with improved features and improved hyperparameters, as an optimized solution.

### Claim 2 (depends on 1)

The computer-implemented method of claim 1, wherein, if the second solution is better than the first solution, updating the first solution with the second solution and recurring the LDS up to given maximum discrepancy.

### Claim 3 (depends on 1)

The computer-implemented method of claim 1, wherein, if the second solution is not better than the first solution, incrementing the discrepancy and recurring the LDS up to the given maximum discrepancy.

### Claim 4 (depends on 1)

The computer-implemented method of claim 1, wherein the discrepancy is a maximum number of modified features and hyperparameters.

### Claim 5 (depends on 1)

The computer-implemented method of claim 1, wherein the updating of the weight table during LDS is enabled by a weighted sum of a number of visits to each feature of the plurality of features and a number of successes for improving an objective value.

### Claim 6 (depends on 5)

The computer-implemented method of claim 5, wherein the weighted sum is given by:
T(x)=w1·v(x)+w2·u(x),
where x is a variable, w1 and w2 are constants, and v(x) and u(x) are the number of visits of successful improvements.

### Claim 7 (depends on 1)

The computer-implemented method of claim 1, wherein the features of the plurality of features are ordered in an ascending order.

### Claim 8

A computer program product for simultaneous feature selection and hyperparameter optimization of non-linear models of machine learning, the computer program product comprising a computer readable storage medium having program instructions embodied therewith, the program instructions executable by a computer to cause the computer to:
set a first solution having first hyperparameters and a first set of features selected from a plurality of features of a training data set;
initialize a weight table providing a score for each feature of the first set of features;
initialize a discrepancy;
perform a limited discrepancy search (LDS), according to an order based on the weight table, to obtain a second solution having second hyperparameters and a second set of features from the plurality of features by swapping the first set of features and switching the first hyperparameters from the first solution with the discrepancy, while updating the weight table during LDS;
compare the second solution with the first solution; and
obtain a new solution with improved features and improved hyperparameters, as an optimized solution.

### Claim 9 (depends on 8)

The computer program product of claim 8, wherein, if the second solution is better than the first solution, updating the first solution with the second solution and recurring the LDS up to given maximum discrepancy.

### Claim 10 (depends on 8)

The computer program product of claim 8, wherein, if the second solution is not better than the first solution, incrementing the discrepancy and recurring the LDS up to the given maximum discrepancy.

### Claim 11 (depends on 8)

The computer program product of claim 8, wherein the discrepancy is a maximum number of modified features and hyperparameters.

### Claim 12 (depends on 8)

The computer program product of claim 8, wherein the updating of the weight table during LDS is enabled by a weighted sum of a number of visits to each feature of the plurality of features and a number of successes for improving an objective value.

### Claim 13 (depends on 12)

The computer program product of claim 12, wherein the weighted sum is given by:
T(x) =w1 v(x) +w2 u(x),
where x is a variable, w1 and w2 are constants, and v(x) and u(x) are the number of visits of successful improvements.

### Claim 14 (depends on 8)

The computer program product of claim 8, wherein the features of the plurality of features are ordered in an ascending order.

### Claim 15

A system for simultaneous feature selection and hyperparameter optimization of non-linear models of machine learning, the system comprising:
a memory; and
one or more processors in communication with the memory configured to:
set a first solution having first hyperparameters and a first set of features selected from a plurality of features of a training data set;
initialize a weight table providing a score for each feature of the first set of features;
initialize a discrepancy;
perform a limited discrepancy search (LDS), according to an order based on the weight table, to obtain a second solution having second hyperparameters and a second set of features from the plurality of features by swapping the first set of features and switching the first hyperparameters from the first solution with the discrepancy, while updating the weight table during LDS;
compare the second solution with the first solution; and
obtain a new solution with improved features and improved hyperparameters, as an optimized solution.

### Claim 16 (depends on 15)

The system of claim 15, wherein, if the second solution is better than the first solution, updating the first solution with the second solution and recurring the LDS up to given maximum discrepancy.

### Claim 17 (depends on 15)

The system of claim 15, wherein, if the second solution is not better than the first solution, incrementing the discrepancy and recurring the LDS up to the given maximum discrepancy.

### Claim 18 (depends on 15)

The system of claim 15, wherein the discrepancy is a maximum number of modified features and hyperparameters.

### Claim 19 (depends on 15)

The system of claim 15, wherein the updating of the weight table during LDS is enabled by a weighted sum of a number of visits to each feature of the plurality of features and a number of successes for improving an objective value.

### Claim 20 (depends on 19)

The system of claim 19, wherein the weighted sum is given by:
T(x)=w1·v(x)+w2·u(x), where x is a variable, w1 and w2 are constants, and v(x) and u(x) are the number of visits of successful improvements.
