# US 12,001,957 B2 · Methods and systems for neural architecture search

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,001,957 B2
- Document ID: US12001957B2
- Status: Granted
- Kind: B2
- Application number: 16/586,363
- Filed: 2019-09-27
- Published: 2020-04-02
- Granted: 2024-06-04
- Earliest claimed: 2018-09-27
- Assignees: Swisscom AG
- Applicants: Swisscom AG
- Inventors: Yassine Benyahia; Kamil Bennani-Smires; Michael Baeriswyl; Claudiu Musat
- Web page: https://patentgrep.com/patent/US12001957B2

## CPC

- G06N 3/086: using evolutionary algorithms, e.g. genetic algorithms or genetic programming
- G06F 18/24: Classification techniques
- G06F 40/20: Natural language analysis (semantic analysis of natural language G06F40/30)
- G06N 3/006: based on simulated virtual individual or collective life forms, e.g. social simulations or particle swarm optimisation [PSO]
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G06N 3/045: Combinations of networks
- G06N 3/048: Activation functions
- G06N 3/0499: Feedforward networks
- G06N 3/082: modifying the architecture, e.g. adding, deleting or silencing nodes or connections
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06N 3/09: Supervised learning
- G06N 3/092: Reinforcement learning
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

Methods and systems are provided for neural architecture search. A computer-implemented neural architecture search may be used for providing a neural network configured to perform a selected task. A computational graph is obtained, which includes a plurality of nodes, edges and weightings associated with the nodes and/or edges. The computational graph includes a plurality of candidate models in the form of subgraphs of the computational graph. Selected subgraphs may be trained sequentially, with the weightings corresponding to each said subgraph being updated in response to training. For each weighting in a subgraph which is shared with another subgraph, updates to the weightings are controlled based on an indication of how important to another subgraph a node/edge associated with that weighting is.

## Claims

### Claim 1

A method comprising:
in a system comprising one or more processing circuits:
determining, by the one or more processing circuits, a preferred model for performing a selected task, wherein the determining comprises:
obtaining a computational graph comprising: a plurality of nodes connected by a plurality of edges, and a plurality of weightings configured to scale input data provided to nodes along edges, wherein each node is configured to:
receive at least one item of input data from a preceding node connected to the node via an edge;
perform an operation on the input data to provide output data, wherein each item of input data is scaled according to a weighting associated with the node and/or edge; and
provide the output data to a subsequent node via an edge in the graph;
wherein the computational graph defines a first model and a second model, each model being a subgraph in the computational graph that has a selection of the plurality of nodes, edges and associated weightings, wherein some of the selected nodes, edges and weightings are shared between both the first model and the second model;
updating the weightings of the first model based on training the first model to perform the selected task;
training the second model using a training loss function which includes a component indicative of a measure for weightings in the second model, wherein the training loss function comprises a scaling factor configured to inhibit an exploding gradient associated with any of the weightings in the second model;
updating the weightings of the second model based on training the second model to perform the same selected task as the first model, wherein:
updating the weightings of the second model comprises updating some of the weightings updated in step which are shared between the first and second models, and
updating the shared weighting is controlled based on an indication of importance for the trained first model a node and/or edge associated with the weighting; and
identifying the preferred model, the identifying comprising a selection of nodes, edges and associated weightings from the computational graph, wherein the preferred model is identified based on an analysis of the first and second trained models;
configuring, by the one or more processing circuits, a neural network to perform the selected task based on the preferred model; and
performing, by the one or more processing circuits, the selected task using the neural network, wherein the selected task comprises one or more of natural language processing, image recognition, classification and/or modelling of physical systems, data processing, and generation of search results.

### Claim 2 (depends on 1)

The method of claim 1, wherein the neural network is configured for at least one of: natural language processing, image recognition, classification and/or modelling of physical systems, data processing and generation of search results.

### Claim 3 (depends on 1)

The method of claim 1, comprising determining the indication of importance for each shared weighting using a measure based on a gradient or higher order derivative of a selected loss function with respect to the weighting.

### Claim 4 (depends on 3)

The method of claim 3, wherein controlling the update to the shared weighting comprises scaling a magnitude of the update based on a value for the measure.

### Claim 5 (depends on 1)

The method of claim 1, comprising initializing weightings of the second model, prior to training of the second model, based on the trained weightings of the first model.

### Claim 6 (depends on 1)

The method of claim 1, wherein identifying the preferred model for the neural network comprises analyzing the first and second models using a policy based on at least one of reinforcement learning and evolutionary learning.

### Claim 7 (depends on 6)

The method of claim 6, wherein identifying the preferred model comprises performing one or more steps that comprise:
determining which model performs best out of the first and second models based on testing both models using a validation data set;
holding the value for the trained weightings constant and updating the policy of based on an analysis of the better performing model;
obtaining based on the updated policy, a suggestion for third and fourth models, wherein some nodes, edges and weightings are shared between both the third model and the fourth model;
updating the weightings of the third model based on training the third model to perform the selected task;
updating the weightings of the fourth model based on training the fourth model to perform the same selected task as the third model, wherein updating the weightings of the fourth model comprises updating some of the weightings which are shared between the third and fourth models, and wherein updating the shared weighting is controlled based on an indication of importance for the trained third model a node and/or edge associated with the weighting is; and
identifying the preferred model based on an analysis of the third and fourth trained models.

### Claim 8 (depends on 6)

The method of claim 6, comprising operating a controller based on the policy, wherein the operating comprises use of a reward based on: a component indicative of a goodness of fit for a model, and a component indicative of complexity of the model.

### Claim 9 (depends on 8)

The method of claim 8, wherein the two components of the reward are separated and/or scaled to prevent the reward being negative.

### Claim 10 (depends on 6)

The method of claim 6, comprising repeating the one or more steps for different suggested models until at least one of: a time threshold is reached, and a difference between a best performing model and a subsequently suggested model is below a threshold difference.

### Claim 11 (depends on 10)

The method of claim 10, comprising selecting the best performing suggested model from a last iteration as the model architecture for the neural network configured to perform the selected task.

### Claim 12 (depends on 1)

The method of claim 1, wherein obtaining a model comprises selecting nodes based on: the selected task, and an indication of a performance of the nodes on similar tasks to the selected task, for example, wherein the nodes comprise at least one of: neurons, and blocks of neurons.

### Claim 13 (depends on 1)

The method of claim 1, comprising repeating the updating the weightings of the first model, the updating the weightings of the second model, and the identifying of the preferred model for the neural network for a selected number of times, wherein initially they are repeated without controlling the updates to weightings.

### Claim 14 (depends on 13)

The method of claim 13, comprising repeating the updating the weightings of the first model, the updating the weightings of the second model, and the identifying of the preferred model for the neural network for a selected number of times initially without controlling the updates to weightings.