# US 2022/0036150 A1 · System and method for synthesis of compact and accurate neural networks (scann)

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2022/0036150 A1
- Document ID: US20220036150A1
- Status: Published application
- Kind: A1
- Application number: 17/275,949
- Filed: 2019-07-12
- Published: 2022-02-03
- Earliest claimed: 2018-09-18
- Assignees: The Trustees of Princeton University
- Applicants: The Trustees of Princeton University
- Inventors: Shayan HASSANTABAR; Zeyu WANG; Niraj K. JHA
- Web page: https://patentgrep.com/patent/US20220036150A1

## CPC

- G06N 3/04: Architecture, e.g. interconnection topology
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/0495: Quantised networks; Sparse networks; Compressed networks
- G06N 3/082: modifying the architecture, e.g. adding, deleting or silencing nodes or connections
- G06N 3/09: Supervised learning
- G06N 20/00: Machine learning
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

According to various embodiments, a method for generating a compact and accurate neural network for a dataset is disclosed. The method includes providing an initial neural network architecture; performing a dataset modification on the dataset, the dataset modification including reducing dimensionality of the dataset; performing a first compression step on the initial neural network architecture that results in a compressed neural network architecture, the first compression step including reducing a number of neurons in one or more layers of the initial neural network architecture based on a feature compression ratio determined by the reduced dimensionality of the dataset; and performing a second compression step on the compressed neural network architecture, the second compression step including one or more of iteratively growing connections, growing neurons, and pruning connections until a desired neural network architecture has been generated.

## Claims

### Claim 1

A method for generating a compact and accurate neural network for a dataset, the method comprising:
providing an initial neural network architecture;
performing a dataset modification on the dataset, the dataset modification comprising reducing dimensionality of the dataset;
performing a first compression step on the initial neural network architecture that results in a compressed neural network architecture, the first compression step comprising reducing a number of neurons in one or more layers of the initial neural network architecture based on a feature compression ratio determined by the reduced dimensionality of the dataset; and
performing a second compression step on the compressed neural network architecture, the second compression step comprising iteratively growing at least one of connections and neurons to increase a size of the compressed neural network architecture and iteratively pruning connections to decrease the size of the compressed neural network architecture until a desired neural network architecture has been generated.

### Claim 2 (depends on 1)

The method of claim 1, further comprising adding one or more hidden layers to the initial neural network architecture.

### Claim 3 (depends on 1)

The method of claim 1, wherein dataset modification further comprises normalizing the dataset.

### Claim 4 (depends on 1)

The method of claim 1, wherein reducing dimensionality of the dataset comprises one or more of random projection, principal component analysis (PCA), polynomial kernel PCA, Gaussian kernel PCA, factor analysis, isomap, independent component analysis, and spectral embedding.

### Claim 5 (depends on 1)

The method of claim 1, wherein the first compression step further comprises computing the feature compression ratio.

### Claim 6

(canceled)

### Claim 7

(canceled)

### Claim 8 (depends on 1)

The method of claim 1, wherein growing connections further comprises growing connections only between adjacent layers.

### Claim 9 (depends on 1)

The method of claim 1, wherein growing connections is based on one of gradient-based growth, full growth, and random growth.

### Claim 10 (depends on 1)

The method of claim 1, wherein growing neurons is based on one of duplication and random addition.

### Claim 11 (depends on 1)

The method of claim 1, wherein pruning connections is based on magnitude-based pruning.

### Claim 12

A system for generating a compact and accurate neural network for a dataset, the system comprising one or more processors configured to:
provide an initial neural network architecture;
perform a dataset modification on the dataset, the dataset modification comprising reducing dimensionality of the dataset;
perform a first compression step on the initial neural network architecture that results in a compressed neural network architecture, the first compression step comprising reducing a number of neurons in one or more layers of the initial neural network architecture based on a feature compression ratio determined by the reduced dimensionality of the dataset; and
perform a second compression step on the compressed neural network architecture, the second compression step comprising iteratively growing at least one of connections and neurons to increase a size of the compressed neural network architecture and iteratively pruning connections to decrease the size of the compressed neural network architecture until a desired neural network architecture has been generated.

### Claim 13 (depends on 12)

The system of claim 12, wherein the one or more processors are further configured to add one or more hidden layers to the initial neural network architecture.

### Claim 14 (depends on 12)

The system of claim 12, wherein dataset modification further comprises normalizing the dataset.

### Claim 15 (depends on 12)

The system of claim 12, wherein reducing dimensionality of the dataset comprises one or more of random projection, principal component analysis (PCA), polynomial kernel PCA, Gaussian kernel PCA, factor analysis, isomap, independent component analysis, and spectral embedding.

### Claim 16 (depends on 12)

The system of claim 12, wherein the first compression step further comprises computing the feature compression ratio.

### Claim 17

(canceled)

### Claim 18

(canceled)

### Claim 19 (depends on 12)

The system of claim 12, wherein growing connections further comprises growing connections only between adjacent layers.

### Claim 20 (depends on 12)

The system of claim 12, wherein growing connections is based on one of gradient-based growth, full growth, and random growth.

### Claim 21 (depends on 12)

The system of claim 12, wherein growing neurons is based on one of duplication and random addition.

### Claim 22 (depends on 12)

The system of claim 12, wherein pruning connections is based on magnitude-based pruning.

### Claim 23

A non-transitory computer-readable medium having stored thereon a computer program for execution by a processor configured to perform a method for generating a compact and accurate neural network for a dataset, the method comprising:
providing an initial neural network architecture;
performing a dataset modification on the dataset, the dataset modification comprising reducing dimensionality of the dataset;
performing a first compression step on the initial neural network architecture that results in a compressed neural network architecture, the first compression step comprising reducing a number of neurons in one or more layers of the initial neural network architecture based on a feature compression ratio determined by the reduced dimensionality of the dataset;
performing a second compression step on the compressed neural network architecture, the second compression step comprising iteratively growing at least one of connections and neurons to increase a size of the compressed neural network architecture and iteratively pruning connections to decrease the size of the compressed neural network architecture until a desired neural network architecture has been generated.

### Claim 24 (depends on 23)

The computer-readable medium of claim 23, wherein the method further comprises adding one or more hidden layers to the initial neural network architecture.

### Claim 25 (depends on 23)

The computer-readable medium of claim 23, wherein dataset modification further comprises normalizing the dataset.

### Claim 26 (depends on 23)

The computer-readable medium of claim 23, wherein reducing dimensionality of the dataset comprises one or more of random projection, principal component analysis (PCA), polynomial kernel PCA, Gaussian kernel PCA, factor analysis, isomap, independent component analysis, and spectral embedding.

### Claim 27 (depends on 23)

The computer-readable medium of claim 23, wherein the first compression step further comprises computing the feature compression ratio.

### Claim 28

(canceled)

### Claim 29

(canceled)

### Claim 30 (depends on 23)

The computer-readable medium of claim 23, wherein growing connections further comprises growing connections only between adjacent layers.

### Claim 31 (depends on 23)

The computer-readable medium of claim 23, wherein growing connections is based on one of gradient-based growth, full growth, and random growth.

### Claim 32 (depends on 23)

The computer-readable medium of claim 23, wherein growing neurons is based on one of duplication and random addition.

### Claim 33 (depends on 23)

The computer-readable medium of claim 23, wherein pruning connections is based on magnitude-based pruning.
