# US 11,443,162 B2 · Resource constrained neural network architecture search

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,443,162 B2
- Document ID: US11443162B2
- Status: Granted
- Kind: B2
- Application number: 16/549,715
- Filed: 2019-08-23
- Published: 2021-02-25
- Granted: 2022-09-13
- Earliest claimed: 2019-08-23
- Assignees: Google Inc.
- Applicants: Google LLC
- Inventors: Ming-Hsuan Yang; Xiaojie Jin; Joshua Foster Slocum; Shengyang Dai; Jiang Wang
- Web page: https://patentgrep.com/patent/US11443162B2

## CPC

- G06N 3/04: Architecture, e.g. interconnection topology
- G06N 3/082: modifying the architecture, e.g. adding, deleting or silencing nodes or connections
- G06F 16/9024: {Graphs; Linked lists (G06F16/9027 takes precedence)}
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/063: using electronic means
- G06N 3/09: Supervised learning
- G06N 3/0985: Hyperparameter optimisation; Meta-learning; Learning-to-learn
- G06N 20/00: Machine learning
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

Methods, and systems, including computer programs encoded on computer storage media for neural network architecture search. A method includes defining a neural network computational cell, the computational cell including a directed graph of nodes representing respective neural network latent representations and edges representing respective operations that transform a respective neural network latent representation; replacing each operation that transforms a respective neural network latent representation with a respective linear combination of candidate operations, where each candidate operation in a respective linear combination has a respective mixing weight that is parameterized by one or more computational cell hyper parameters; iteratively adjusting values of the computational cell hyper parameters and weights to optimize a validation loss function subject to computational resource constraints; and generating a neural network for performing a machine learning task using the defined computational cell and the adjusted values of the computational cell hyper parameters and weights.

## Claims

### Claim 1

A computer implemented method comprising:
defining a computational cell of a neural network, wherein the computational cell comprises a directed graph of nodes and edges, each node representing a respective neural network latent representation and each edge representing a respective operation that transforms a respective neural network latent representation;
replacing each operation that transforms a respective neural network latent representation with a respective linear combination of candidate operations from a predefined set of candidate operations, wherein each candidate operation in a respective linear combination has a respective mixing weight that is parameterized by one or more computational cell hyper parameters;
iteratively adjusting values of i) the computational cell hyper parameters, and ii) computational cell weights, to optimize a validation loss function subject to one or more computational resource constraints, comprising, for each iteration:
performing an unconstrained optimization of the validation loss function to update values of the computational cell hyper parameters for a previous iteration and to obtain adjusted values of the computational cell weights; and
projecting the updated values of the computational cell hyper parameters to a nearest point in a feasible set defined by the one or more resource constraints to obtain adjusted values of the computational cell hyper parameters; and
generating a neural network for performing a machine learning task using the defined computational cell and the adjusted values of the computational cell hyper parameters and computational cell weights.

### Claim 2 (depends on 1)

The method of claim 1, wherein generating a neural network for performing a machine learning task using the defined computational cell and the adjusted values of the computational cell hyper parameters and computational cell weights comprises:
defining a discrete computational cell architecture by replacing each linear combination of candidate operations with a single operation with a largest mixing weight that is parameterized by one or more adjusted computational cell hyper parameters; and
generating a neural network for performing a machine learning task using the defined discrete computational cell architecture and the adjusted values of the computational cell weights.

### Claim 3 (depends on 1)

The method of claim 1, wherein generating a neural network for performing a machine learning task using the defined computational cell and the adjusted values of the computational cell hyper parameters and computational cell weights comprises stacking multiple copies of the defined computational cell, wherein each copy has a same cell architecture defined by the adjusted values of the computational cell hyper parameters.

### Claim 4 (depends on 3)

The method of claim 3, further comprising:
generating the multiple copies of the defined computational cell; and
training each generated copy of the defined computational cell on respective training data.

### Claim 5 (depends on 3)

The method of claim 3, wherein stacking multiple copies of the defined computational cell comprises interleaving one or more additional neural network layers between the copies of the defined computational cell.

### Claim 6 (depends on 5)

The method of claim 5, wherein the one or more additional neural network layers comprises a connection computational cell that includes one input node and one intermediate node, and wherein the method further comprises learning the connection cell by iteratively adjusting values of i) the connection computational cell hyper parameters, and ii) connection computational cell weights, to optimize the validation loss function subject to one or more computational resource constraints.

### Claim 7 (depends on 1)

The method of claim 1, wherein the validation loss function represents a measure of error obtained after running a validation dataset through the defined computational set.

### Claim 8 (depends on 1)

The method of claim 1, wherein iteratively adjusting values of i) the computational cell hyper parameters, and ii) computational cell weights, to optimize a validation loss function comprises performing a bi-level optimization of the validation loss function and a training loss function that represents a measure of error obtained on training data, wherein the computational cell hyper parameters comprise upper level parameters and the computational cell weights comprise lower level parameters.

### Claim 9 (depends on 1)

The method of claim 1, wherein the one or more computational resource constraints comprise user defined constraints on one or more of memory, number of float point operations, or inference speed.

### Claim 10 (depends on 1)

The method of claim 1, wherein iteratively adjusting values of i) the computational cell hyper parameters, and ii) computational cell weights, to optimize a validation loss function subject to one or more computational resource constraints comprises:
defining a respective cost function for each computational resource constraint, wherein each defined cost function maps the computational cell hyper parameters to a respective resource cost.

### Claim 11 (depends on 10)

The method of claim 10, wherein a respective resource cost of an edge in the computational cell is calculated as a softmax over the costs of operations in the candidate set of operations.

### Claim 12 (depends on 10)

The method of claim 10, further comprising setting lower and higher bound constraints for each defined cost function.

### Claim 13 (depends on 1)

The method of claim 1, wherein performing an unconstrained optimization of the validation loss function to update values of the computational cell hyper parameters for a previous iteration and to obtain adjusted values of the computational cell weights comprises iteratively performing the unconstrained optimization of the validation loss function, comprising, for each iteration:
obtaining values of the computational cell hyper parameters and computational cell weights for the iteration, comprising obtaining randomly initialized values of the computational cell hyper parameters and the computational cell weights for a first iteration or obtaining values of the computational cell hyper parameters and the computational cell weights from a previous iteration;
iteratively, for a predetermined number of iterations or until predefined convergence criteria are met:
fixing the obtained values of the computational cell hyper parameters for the iteration and obtaining updated computational cell weights based on a gradient of a training loss function with respect to the computational cell weights; and
fixing the updated computational cell weights and obtaining the updated values of the computational cell hyper parameters based on a gradient of the validation loss function with respect to the computational cell hyper parameters.

### Claim 14 (depends on 13)

The method of claim 13, wherein fixing the updated computational cell weights and obtaining updated computational cell hyper parameters based on a gradient of the validation loss function with respect to the computational cell hyper parameters comprises assuming that the computational cell hyper parameters and the computational cell weights are independent.

### Claim 15 (depends on 1)

The method of claim 1, wherein projecting the updated values of the computational cell hyper parameters to a nearest point in a feasible set defined by the one or more resource constraints comprises identifying an element in the feasible set that minimizes a 2-norm of the difference between i) the updated values of the computational cell hyper parameters, and ii) the element.

### Claim 16 (depends on 1)

The method of claim 1, wherein the predefined set of candidate operations comprises pooling operations, convolutional operations or connection operations.

### Claim 17 (depends on 1)

The method of claim 1, further comprising:
defining multiple computational cells of the neural network, wherein each defined computational cell can be represented by a respective directed graph of nodes and edges;
for each defined computational cell of the multiple defined computational cells:
replacing each operation that transforms a respective neural network latent representation with a respective linear combination of candidate operations from a predefined set of candidate operations, wherein each candidate operation in a respective linear combination has a respective mixing weight that is parameterized by one or more computational cell hyper parameters;
iteratively adjusting values of i) the computational cell hyper parameters, and ii) computational cell weights, to optimize a validation loss function subject to one or more computational resource constraints, comprising, for each iteration:
performing an unconstrained optimization of the validation loss function to update values of the computational cell hyper parameters for a previous iteration and to obtain adjusted values of the computational cell weights; and
projecting the updated values of the computational cell hyper parameters to a nearest point in a feasible set defined by the one or more resource constraints to obtain adjusted values of the computational cell hyper parameters; and
generating a neural network for performing a machine learning task using the defined multiple computational cells and the adjusted values of the respective computational cell hyper parameters and computational cell weights.

### Claim 18 (depends on 1)

The method of claim 1, further comprising:
training the generated neural network on training data to obtain a trained neural network; and
performing the machine learning task using the trained neural network.

### Claim 19

A system comprising one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising:
defining a computational cell of a neural network, wherein the computational cell comprises a directed graph of nodes and edges, each node representing a respective neural network latent representation and each edge representing a respective operation that transforms a respective neural network latent representation;
replacing each operation that transforms a respective neural network latent representation with a respective linear combination of candidate operations from a predefined set of candidate operations, wherein each candidate operation in a respective linear combination has a respective mixing weight that is parameterized by one or more computational cell hyper parameters;
iteratively adjusting values of i) the computational cell hyper parameters, and ii) computational cell weights, to optimize a validation loss function subject to one or more computational resource constraints, comprising, for each iteration:
performing an unconstrained optimization of the validation loss function to update values of the computational cell hyper parameters for a previous iteration and to obtain adjusted values of the computational cell weights; and
projecting the updated values of the computational cell hyper parameters to a nearest point in a feasible set defined by the one or more resource constraints to obtain adjusted values of the computational cell hyper parameters; and
generating a neural network for performing a machine learning task using the defined computational cell and the adjusted values of the computational cell hyper parameters and computational cell weights.

### Claim 20

A computer-readable storage medium comprising instructions stored thereon that are executable by a processing device and upon such execution cause the processing device to perform operations comprising:
defining a computational cell of a neural network, wherein the computational cell comprises a directed graph of nodes and edges, each node representing a respective neural network latent representation and each edge representing a respective operation that transforms a respective neural network latent representation;
replacing each operation that transforms a respective neural network latent representation with a respective linear combination of candidate operations from a predefined set of candidate operations, wherein each candidate operation in a respective linear combination has a respective mixing weight that is parameterized by one or more computational cell hyper parameters;
iteratively adjusting values of i) the computational cell hyper parameters, and ii) computational cell weights, to optimize a validation loss function subject to one or more computational resource constraints, comprising, for each iteration:
performing an unconstrained optimization of the validation loss function to update values of the computational cell hyper parameters for a previous iteration and to obtain adjusted values of the computational cell weights; and
projecting the updated values of the computational cell hyper parameters to a nearest point in a feasible set defined by the one or more resource constraints to obtain adjusted values of the computational cell hyper parameters; and
generating a neural network for performing a machine learning task using the defined computational cell and the adjusted values of the computational cell hyper parameters and computational cell weights.