# US 2019/0294999 A1 · Selecting hyper parameters for machine learning algorithms based on past training results

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2019/0294999 A1
- Document ID: US20190294999A1
- Status: Published application
- Kind: A1
- Application number: 16/441,150
- Filed: 2019-06-14
- Published: 2019-09-26
- Earliest claimed: 2018-06-16
- Inventors: Moshe Guttmann
- Web page: https://patentgrep.com/patent/US20190294999A1

## CPC

- G06N 5/025: Extracting rules from data
- G06F 3/04845: for image manipulation, e.g. dragging, rotation, expansion or change of colour
- G06F 18/217: {Validation; Performance evaluation; Active pattern learning techniques}
- G06F 18/231: Hierarchical techniques, i.e. dividing or merging pattern sets so as to obtain a dendrogram
- G06F 18/29: {Graphical models, e.g. Bayesian networks}
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/08: Learning methods
- G06N 3/082: modifying the architecture, e.g. adding, deleting or silencing nodes or connections
- G06N 3/09: Supervised learning
- G06N 3/091: Active learning
- G06N 3/0985: Hyperparameter optimisation; Meta-learning; Learning-to-learn
- G06N 5/04: Inference or reasoning models
- G06N 5/043: Distributed expert systems; Blackboards
- G06N 5/047: Pattern matching networks; Rete networks
- G06N 20/00: Machine learning
- G06N 20/10: using kernel methods, e.g. support vector machines [SVM]
- G06N 20/20: Ensemble learning
- G06V 10/94: Hardware or software architectures specially adapted for image or video understanding
- G06V 20/20: in augmented reality scenes
- G06V 20/70: Labelling scene content, e.g. deriving syntactic or semantic representations
- G06V 40/10: Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G10L 15/065: Adaptation
- G10L 15/00: Speech recognition (G10L17/00 takes precedence)
- G10L 17/00: Speaker identification or verification techniques
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING

## Abstract

Systems and methods for selecting hyper parameters for machine learning algorithms based on past training results are provided. For example, groups of values of hyper parameters may be obtained. Further, in some examples, results of training the machine learning algorithm using different pluralities of training examples and/or the different group of values of hyper parameters may be obtained. Further, in some examples, the results and the groups of values of hyper parameters may be used to select at least one value of a hyper parameter for a prospective training of the machine learning algorithm.

## Claims

### Claim 1

A method for selecting hyper parameters for machine learning algorithms based on past training results, the method comprising:
obtaining a first group of values of hyper parameters, a second group of values of hyper parameters and a third group of values of hyper parameters, the second group of values of hyper parameters differs from the first group of values of hyper parameters, and the third group of values of hyper parameters differs from the first group of values of hyper parameters and the second group of values of hyper parameters;
obtaining a first result of training a machine learning algorithm using a first plurality of training examples and the first group of values of hyper parameters;
obtaining a second result of training the machine learning algorithm using a second plurality of training examples and the second group of values of hyper parameters;
obtaining a third result of training the machine learning algorithm using a third plurality of training examples and the third group of values of hyper parameters; and
using the first result, the second result, the third result, the first group of values of hyper parameters, the second group of values of hyper parameters and the third group of values of hyper parameters to select at least one value of a hyper parameter for a prospective training of the machine learning algorithm.

### Claim 2 (depends on 1)

The method of claim 1, wherein the first group of values of hyper parameters comprises a first learning rate value, the second group of values of hyper parameters comprises a second learning rate value, the third group of values of hyper parameters comprises a third learning rate value, and further comprising using the first result, the second result, the third result, the first learning rate value, the second learning rate value and the third learning rate value to select a learning rate value for the prospective training of the machine learning algorithm.

### Claim 3 (depends on 1)

The method of claim 1, wherein the first group of values of hyper parameters comprises a first selection of a learning rate update method, the second group of values of hyper parameters comprises a second selection of a learning rate update method, the third group of values of hyper parameters comprises a third selection of a learning rate update method, and further comprising using the first result, the second result, the third result, the first selection of a learning rate update method, the second selection of a learning rate update method and the third selection of a learning rate update method to select a learning rate update method for the prospective training of the machine learning algorithm.

### Claim 4 (depends on 1)

The method of claim 1, wherein the first group of values of hyper parameters comprises a first batch size, the second group of values of hyper parameters comprises a second batch size, the third group of values of hyper parameters comprises a third batch size, and further comprising using the first result, the second result, the third result, the first batch size, the second batch size and the third batch size to select a batch size for the prospective training of the machine learning algorithm.

### Claim 5 (depends on 1)

The method of claim 1, wherein the first group of values of hyper parameters comprises a first network structure, the second group of values of hyper parameters comprises a second network structure, the third group of values of hyper parameters comprises a third network structure, and further comprising using the first result, the second result, the third result, the first network structure, the second network structure and the third network structure to select a network structure for the prospective training of the machine learning algorithm.

### Claim 6 (depends on 1)

The method of claim 1, wherein the first group of values of hyper parameters comprises a first number of layers, the second group of values of hyper parameters comprises a second number of layers, the third group of values of hyper parameters comprises a third number of layers, and further comprising using the first result, the second result, the third result, the first number of layers, the second number of layers and the third number of layers to select a number of layers for the prospective training of the machine learning algorithm.

### Claim 7 (depends on 1)

The method of claim 1, wherein the first group of values of hyper parameters comprises a first number of artificial neurons, the second group of values of hyper parameters comprises a second number of artificial neurons, the third group of values of hyper parameters comprises a third number of artificial neurons, and further comprising using the first result, the second result, the third result, the first number of artificial neurons, the second number of artificial neurons and the third number of artificial neurons to select a number of artificial neurons for the prospective training of the machine learning algorithm.

### Claim 8 (depends on 1)

The method of claim 1, wherein the first group of values of hyper parameters comprises a first selection of an activation function, the second group of values of hyper parameters comprises a second selection of an activation function, the third group of values of hyper parameters comprises a third selection of an activation function, and further comprising using the first result, the second result, the third result, the first selection of an activation function, the second selection of an activation function and the third selection of an activation function to select an activation function for the prospective training of the machine learning algorithm.

### Claim 9 (depends on 1)

The method of claim 1, wherein the first group of values of hyper parameters comprises a first selection of a loss function, the second group of values of hyper parameters comprises a second selection of a loss function, the third group of values of hyper parameters comprises a third selection of a loss function, and further comprising using the first result, the second result, the third result, the first selection of a loss function, the second selection of a loss function and the third selection of a loss function to select a loss function for the prospective training of the machine learning algorithm.

### Claim 10 (depends on 1)

The method of claim 1, wherein the first group of values of hyper parameters comprises a first selection of an initialization method, the second group of values of hyper parameters comprises a second selection of an initialization method, the third group of values of hyper parameters comprises a third selection of an initialization method, and further comprising using the first result, the second result, the third result, the first selection of an initialization method, the second selection of an initialization method and the third selection of an initialization method to select an initialization method for the prospective training of the machine learning algorithm.

### Claim 11 (depends on 1)

The method of claim 1, wherein the first group of values of hyper parameters comprises a first selection of a kernel function, the second group of values of hyper parameters comprises a second selection of a kernel function, the third group of values of hyper parameters comprises a third selection of a kernel function, and further comprising using the first result, the second result, the third result, the first selection of a kernel function, the second selection of a kernel function and the third selection of a kernel function to select a kernel function for the prospective training of the machine learning algorithm.

### Claim 12 (depends on 1)

The method of claim 1, wherein the first group of values of hyper parameters comprises a first selection of a distance measure, the second group of values of hyper parameters comprises a second selection of a distance measure, the third group of values of hyper parameters comprises a third selection of a distance measure, and further comprising using the first result, the second result, the third result, the first selection of a distance measure, the second selection of a distance measure and the third selection of a distance measure to select a distance measure for the prospective training of the machine learning algorithm.

### Claim 13 (depends on 1)

The method of claim 1, wherein the first group of values of hyper parameters comprises a first selection of a stopping condition, the second group of values of hyper parameters comprises a second selection of a stopping condition, the third group of values of hyper parameters comprises a third selection of a stopping condition, and further comprising using the first result, the second result, the third result, the first selection of a stopping condition, the second selection of a stopping condition and the third selection of a stopping condition to select a stopping condition for the prospective training of the machine learning algorithm.

### Claim 14 (depends on 1)

The method of claim 1, wherein the first group of values of hyper parameters comprises a first sampling criterion used to sample the first plurality of training examples from a base group of training examples, the second group of values of hyper parameters comprises a second sampling criterion used to sample the second plurality of training examples from the base group of training examples, the third group of values of hyper parameters comprises a third sampling criterion used to sample the third plurality of training examples from a base group of training examples, and further comprising using the first result, the second result, the third result, the first sampling criterion, the second sampling criterion and the third sampling criterion to select a sampling criterion for sampling a plurality of training examples from the base group of training examples for the prospective training of the machine learning algorithm.

### Claim 15 (depends on 1)

The method of claim 1, wherein further comprising:
training the machine learning algorithm using the first plurality of training examples and the first group of values of hyper parameters to generate a first inference model;
using the first inference model and a plurality of testing examples to generate a first plurality of outputs;
comparing the first plurality of outputs and a plurality of desired results to determine the first result.

### Claim 16 (depends on 1)

The method of claim 1, wherein further comprising:
training the machine learning algorithm using the first plurality of training examples and the first group of values of hyper parameters to generate a first inference model;
using the first inference model and a plurality of testing examples to generate a first plurality of outputs;
providing information based on the first plurality of outputs to at least one user;
receiving one or more feedbacks related to the information based on the first plurality of outputs from the at least one user; and
using the one or more feedbacks related to the information based on the first plurality of outputs to determine the first result.

### Claim 17 (depends on 1)

The method of claim 1, wherein further comprising:
determining an embedding in a mathematical space of at least a value of the first group of values of hyper parameters, a value of the second group of values of hyper parameters and a value of the third group of values of hyper parameters; and
using the determined embedding of the values in the mathematical space to select the at least one value of the hyper parameter for the prospective training of the machine learning algorithm.

### Claim 18 (depends on 17)

The method of claim 17, further comprising:
determining whether the first result is preferable to the second result and the third result;
in response to the determination that the first result is preferable to the second result and the third result, selecting the at least one value of the hyper parameter for the prospective training of the machine learning algorithm to be closer to the first group of values of hyper parameters than to the second group of values of hyper parameters and to the third group of values of hyper parameters according to the embedding in the mathematical space; and
in response to the determination that the first result is not preferable to the second result and the third result, selecting the at least one value of the hyper parameter for the prospective training of the machine learning algorithm to be farther from the first group of values of hyper parameters than from at least one of the second group of values of hyper parameters and the third group of values of hyper parameters according to the embedding in the mathematical space.

### Claim 19

A system for selecting hyper parameters for machine learning algorithms based on past training results, the system comprising:
at least one processor configured to:
obtain a first group of values of hyper parameters, a second group of values of hyper parameters and a third group of values of hyper parameters, the second group of values of hyper parameters differs from the first group of values of hyper parameters, and the third group of values of hyper parameters differs from the first group of values of hyper parameters and the second group of values of hyper parameters;
obtain a first result of training a machine learning algorithm using a first plurality of training examples and the first group of values of hyper parameters;
obtain a second result of training the machine learning algorithm using a second plurality of training examples and the second group of values of hyper parameters;
obtain a third result of training the machine learning algorithm using a third plurality of training examples and the third group of values of hyper parameters; and
use the first result, the second result, the third result, the first group of values of hyper parameters, the second group of values of hyper parameters and the third group of values of hyper parameters to select at least one value of a hyper parameter for a prospective training of the machine learning algorithm.

### Claim 20

A non-transitory computer readable medium storing data and computer implementable instructions for carrying out a method for selecting hyper parameters for machine learning algorithms based on past training results, the method comprising:
obtaining a first group of values of hyper parameters, a second group of values of hyper parameters and a third group of values of hyper parameters, the second group of values of hyper parameters differs from the first group of values of hyper parameters, and the third group of values of hyper parameters differs from the first group of values of hyper parameters and the second group of values of hyper parameters;
obtaining a first result of training a machine learning algorithm using a first plurality of training examples and the first group of values of hyper parameters;
obtaining a second result of training the machine learning algorithm using a second plurality of training examples and the second group of values of hyper parameters;
obtaining a third result of training the machine learning algorithm using a third plurality of training examples and the third group of values of hyper parameters; and
using the first result, the second result, the third result, the first group of values of hyper parameters, the second group of values of hyper parameters and the third group of values of hyper parameters to select at least one value of a hyper parameter for a prospective training of the machine learning algorithm.
