# US 2023/0274150 A1 · Performing Inference And Training Using Sparse Neural Network

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2023/0274150 A1
- Document ID: US20230274150A1
- Status: Published application
- Kind: A1
- Application number: 18/312,011
- Filed: 2023-05-04
- Published: 2023-08-31
- Earliest claimed: 2019-11-26
- Assignees: Numenta, Inc.
- Applicants: Numenta, Inc.
- Inventors: Subutai Ahmad; Luiz Scheinkman
- Web page: https://patentgrep.com/patent/US20230274150A1

## CPC

- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06N 3/04: Architecture, e.g. interconnection topology
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/0495: Quantised networks; Sparse networks; Compressed networks
- G06N 3/082: modifying the architecture, e.g. adding, deleting or silencing nodes or connections
- G06N 3/09: Supervised learning
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

An inference system trains and performs inference using a sparse neural network. The sparse neural network may include one or more layers, and each layer may be associated with a set of sparse weights that represent sparse connections between nodes of a layer and nodes of a previous layer. A layer output may be generated by applying the set of sparse weights associated with the layer to the layer output of a previous layer. Moreover, the one or more layers of the sparse neural network may generate sparse layer outputs. By using sparse representations of weights and layer outputs, robustness and stability of the neural network can be significantly improved, while maintaining competitive accuracy.

## Claims

### Claim 1

A method, comprising:
receiving input data by a neural network including a plurality of layers of nodes including a current layer and a previous layer;
generating a layer output of the previous layer based at least on the input data;
applying, to the layer output of the previous layer, a set of weights that represent connections between nodes of the current layer and nodes of the previous layer;
generating intermediate outputs for the current layer by applying the set of weights for the current layer to a layer output of the previous layer; and
generating a layer output for nodes of the current layer by processing the intermediate outputs, the layer output for the current layer generated by intermediate outputs for a subset of selected nodes of the current layer.

### Claim 2 (depends on 1)

The method of claim 1, wherein the selected nodes are nodes having intermediate outputs above a threshold value or within a percentage of highest values.

### Claim 3 (depends on 2)

The method of claim 2, further comprising zeroing intermediate outputs of non-selected nodes but retaining the intermediate outputs of the subset of selected nodes.

### Claim 4 (depends on 1)

The method of claim 1, wherein a sparsity of the layer output for the current layer is equal to or above a threshold.

### Claim 5 (depends on 1)

The method of claim 1, wherein the set of weights have a sparsity equal to or above another threshold.

### Claim 6 (depends on 1)

The method of claim 1, further comprising backpropagating error terms to update the set of weights, wherein the error terms are derived from the layer output and training data corresponding to the input data.

### Claim 7 (depends on 6)

The method of claim 6, further comprising determining a boosting term for each node of the current layer, the boosting term indicating how frequently the node was selected during previous iterations.

### Claim 8 (depends on 7)

The method of claim 7, wherein the boosting term increases as a duty cycle of each node increases and decreases as the duty cycle of each node decreases.

### Claim 9 (depends on 1)

The method of claim 1, wherein the neural network is a convolution neural network.

### Claim 10 (depends on 1)

The method of claim 1, wherein the neural network is implemented on field-programmable gate arrays.

### Claim 11

A non-transitory computer readable storage medium storing instructions thereon, the instructions when executed by a processor cause the processor to:
receive input data by a neural network including a plurality of layers of nodes including a current layer and a previous layer;
generate a layer output of the previous layer based at least on the input data;
apply, to the layer output of the previous layer, a set of weights that represent connections between nodes of the current layer and nodes of the previous layer;
generate intermediate outputs for the current layer by applying the set of weights for the current layer to a layer output of the previous layer; and
generate a layer output for nodes of the current layer by processing the intermediate outputs, the layer output for the current layer generated by intermediate outputs for a subset of selected nodes of the current layer.

### Claim 12 (depends on 11)

The non-transitory computer readable storage medium of claim 11, wherein the selected nodes are nodes having intermediate outputs above a threshold value or within a percentage of highest values.

### Claim 13 (depends on 12)

The non-transitory computer readable storage medium of claim 12, further comprising zeroing intermediate outputs of non-selected nodes but retaining the intermediate outputs of the subset of selected nodes.

### Claim 14 (depends on 11)

The non-transitory computer readable storage medium of claim 11, wherein a sparsity of the layer output for the current layer is equal to or above a threshold.

### Claim 15 (depends on 11)

The non-transitory computer readable storage medium of claim 11, wherein the set of weights have a sparsity equal to or above another threshold.

### Claim 16 (depends on 11)

The non-transitory computer readable storage medium of claim 11, further comprising instructions to backpropagate error terms to update the set of weights, wherein the error terms are derived from the layer output and training data corresponding to the input data.

### Claim 17 (depends on 16)

The non-transitory computer readable storage medium of claim 16, further comprising determining a boosting term for each node of the current layer, the boosting term indicating how frequently the node was selected during previous iterations.

### Claim 18 (depends on 17)

The non-transitory computer readable storage medium of claim 17, wherein the boosting term increases as a duty cycle of each node increases and decreases as the duty cycle of each node decreases.

### Claim 19 (depends on 11)

The non-transitory computer readable storage medium of claim 11, wherein the neural network is a convolution neural network.

### Claim 20

A non-transitory storage medium storing a digital representation of a neural network, the neural network generated by:
receiving input data by a neural network including a plurality of layers of nodes including a current layer and a previous layer;
generating a layer output of the previous layer based at least on the input data;
applying, to the layer output of the previous layer, a set of weights that represent connections between nodes of the current layer and nodes of the previous layer;
generating intermediate outputs for the current layer by applying the set of weights for the current layer to a layer output of the previous layer;
generating a layer output for nodes of the current layer by processing the intermediate outputs, the layer output for the current layer generated by intermediate outputs for a subset of selected nodes of the current layer; and
backpropagating error terms to update the set of weights, wherein the error terms are derived from the layer output and training data corresponding to the input data.
