# US 2022/0027739 A1 · Search space exploration for deep learning

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2022/0027739 A1
- Document ID: US20220027739A1
- Status: Published application
- Kind: A1
- Application number: 16/935,445
- Filed: 2020-07-22
- Published: 2022-01-27
- Earliest claimed: 2020-07-22
- Assignees: International Business Machines Corporation
- Applicants: International Business Machines Corporation
- Inventors: Chao Xue; Yonggang Hu; Lin Dong; Ke Wei Sun
- Web page: https://patentgrep.com/patent/US20220027739A1

## CPC

- G06N 3/086: using evolutionary algorithms, e.g. genetic algorithms or genetic programming
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/082: modifying the architecture, e.g. adding, deleting or silencing nodes or connections
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06N 3/09: Supervised learning
- G06N 3/092: Reinforcement learning
- G06N 3/0985: Hyperparameter optimisation; Meta-learning; Learning-to-learn
- G06N 5/01: Dynamic search techniques; Heuristics; Dynamic trees; Branch-and-bound
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

Aspects of the invention include systems and methods to obtain meta features of a dataset for training in a deep learning application. A method includes selecting an initial search space that defines a type of deep learning architecture representation that specifies hyperparameters for two or more neural network architectures. The method also includes applying a search strategy to the initial search space. One of the two or more neural network architectures are selected based on a result of an evaluation according to the search strategy. A new search space is generated with new hyperparameters using an evolutionary algorithm and a mutation type that defines one or more changes in the hyperparameters specified by the initial search space, and, based on the mutation type, the new hyperparameters are applied to the one of the two or more neural networks or the search strategy is applied to the new search space.

## Claims

### Claim 1

A computer-implemented method comprising:
obtaining, using a processor, meta features corresponding with a dataset configured to be used for training in a deep learning application;
selecting, using the processor, an initial search space, wherein the initial search space defines a type of deep learning architecture representation to represent two or more neural network architectures and specifies hyperparameters for the two or more neural network architectures;
applying, using the processor, a search strategy to the initial search space, wherein the search strategy performs an evaluation of each of the two or more neural network architectures represented according to the initial search space;
selecting, using the processor, one of the two or more neural network architectures based on a result of the evaluation according to the search strategy;
generating, using the processor, a new search space with new hyperparameters that differ from the hyperparameters specified by the initial search space using an evolutionary algorithm and a mutation type, wherein the mutation type defines one or more changes in the hyperparameters specified by the initial search space; and
based on the mutation type, applying the new hyperparameters to the one of the two or more neural networks or applying the search strategy to the new search space.

### Claim 2 (depends on 1)

The computer-implemented method according to claim 1, wherein the selecting the initial search space is based on a supervised learning process.

### Claim 3 (depends on 1)

The computer-implemented method according to claim 1 further comprising selecting the search strategy based on a supervised learning process.

### Claim 4 (depends on 1)

The computer-implemented method according to claim 1, wherein the applying the search strategy to the new search space is based on the mutation type being a change in at least one of a set of operations that include separable convolution, average pooling, max pooling, skip connect, and dilated convolution.

### Claim 5 (depends on 1)

The computer-implemented method according to claim 1, wherein the applying the new hyperparameters to the one of the two or more neural networks is based on the mutation type being a change in a number of layers of the one of the two or more neural network architectures.

### Claim 6 (depends on 1)

The computer-implemented method according to claim 1, wherein the applying the new hyperparameters to the one of the two or more neural networks is based on the mutation type being a change in a minimal number of channels of a convolutional operation in the one of the two or more neural networks.

### Claim 7 (depends on 1)

The computer-implemented method according to claim 1, wherein the applying the new hyperparameters to the one of the two or more neural networks is based on the mutation type being a change in a skip pattern that defines connections between operations in a chain when the deep learning architecture representation is a chain-structure representation.

### Claim 8 (depends on 1)

The computer-implemented method according to claim 1, wherein the applying the new hyperparameters to the one of the two or more neural networks is based on the mutation type being a change in a number of input connections to a node when the deep learning architecture representation is a cell structure representation.

### Claim 9

A system comprising:
a memory having computer readable instructions; and
one or more processors for executing the computer readable instructions, the computer readable instructions controlling the one or more processors to perform operations comprising:
obtaining meta features corresponding with a dataset configured to be used for training in a deep learning application;
selecting an initial search space, wherein the initial search space defines a type of deep learning architecture representation to represent two or more neural network architectures and specifies hyperparameters for the two or more neural network architectures;
applying a search strategy to the initial search space, wherein the search strategy performs an evaluation of each of the two or more neural network architectures represented according to the initial search space;
selecting one of the two or more neural network architectures based on a result of the evaluation according to the search strategy;
generating a new search space with new hyperparameters that differ from the hyperparameters specified by the initial search space using an evolutionary algorithm and a mutation type, wherein the mutation type defines one or more changes in the hyperparameters specified by the initial search space; and
based on the mutation type, applying the new hyperparameters to the one of the two or more neural networks or applying the search strategy to the new search space.

### Claim 10 (depends on 9)

The system according to claim 9, wherein the one or more processors are configured to select the initial search space is based on a supervised learning process.

### Claim 11 (depends on 9)

The system according to claim 9 further comprising the one or more processors selecting the search strategy based on a supervised learning process.

### Claim 12 (depends on 9)

The system according to claim 9, wherein the one or more processors are configured to apply the search strategy to the new search space based on the mutation type being a change in at least one of a set of operations that include separable convolution, average pooling, max pooling, skip connect, and dilated convolution.

### Claim 13 (depends on 9)

The system according to claim 9, wherein the one or more processors are configured to apply the new hyperparameters to the one of the two or more neural networks based on the mutation type being a change in a number of layers of the one of the two or more neural network architectures.

### Claim 14 (depends on 9)

The system according to claim 9, wherein the one or more processors are configured to apply the new hyperparameters to the one of the two or more neural networks based on the mutation type being a change in a minimal number of channels of a convolutional operation in the one of the two or more neural networks.

### Claim 15 (depends on 9)

The system according to claim 9, wherein the one or more processors are configured to apply the new hyperparameters to the one of the two or more neural networks based on the mutation type being a change in a skip pattern that defines connections between operations in a chain when the deep learning architecture representation is a chain-structure representation.

### Claim 16 (depends on 9)

The system according to claim 9, wherein the one or more processors are configured to apply the new hyperparameters to the one of the two or more neural networks based on the mutation type being a change in a number of input connections to a node when the deep learning architecture representation is a cell structure representation.

### Claim 17

A computer program product comprising a computer readable storage medium having program instructions embodied therewith, the program instructions executable by a processor to cause the processor to perform operations comprising:
obtaining, using a processor, meta features corresponding with a dataset configured to be used for training in a deep learning application;
selecting, using the processor, an initial search space, wherein the initial search space defines a type of deep learning architecture representation to represent two or more neural network architectures and specifies hyperparameters for the two or more neural network architectures;
applying, using the processor, a search strategy to the initial search space, wherein the search strategy performs an evaluation of each of the two or more neural network architectures represented according to the initial search space;
selecting, using the processor, one of the two or more neural network architectures based on a result of the evaluation according to the search strategy;
generating, using the processor, a new search space with new hyperparameters that differ from the hyperparameters specified by the initial search space using an evolutionary algorithm and a mutation type, wherein the mutation type defines one or more changes in the hyperparameters specified by the initial search space; and
based on the mutation type, applying the new hyperparameters to the one of the two or more neural networks or applying the search strategy to the new search space.

### Claim 18 (depends on 17)

The computer program product according to claim 17, wherein the selecting the initial search space is based on a supervised learning process, and performing the operations further comprises selecting the search strategy based on a supervised learning process.

### Claim 19 (depends on 17)

The computer program product according to claim 17, wherein the applying the search strategy to the new search space is based on the mutation type being a change in at least one of a set of operations that include separable convolution, average pooling, max pooling, skip connect, and dilated convolution.

### Claim 20 (depends on 17)

The computer program product according to claim 17, wherein the applying the new hyperparameters to the one of the two or more neural networks is based on the mutation type being a change in a number of layers of the one of the two or more neural network architectures, a change in a minimal number of channels of a convolutional operation in the one of the two or more neural networks, a change in a skip pattern that defines connections between operations in a chain when the deep learning architecture representation is a chain-structure representation, or a change in a number of input connections to a node when the deep learning architecture representation is a cell structure representation.
