# US 2022/0067525 A1 · Techniques for pruning neural networks

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2022/0067525 A1
- Document ID: US20220067525A1
- Status: Published application
- Kind: A1
- Application number: 17/002,660
- Filed: 2020-08-25
- Published: 2022-03-03
- Earliest claimed: 2020-08-25
- Assignees: NVIDIA Corporation
- Applicants: NVIDIA Corporation
- Inventors: Dilip Sequeira; Pavlo Molchanov; Gregory Heinrich; Edvard Olav Valter Fagerholm
- Web page: https://patentgrep.com/patent/US20220067525A1

## CPC

- G06N 3/082: modifying the architecture, e.g. adding, deleting or silencing nodes or connections
- B60W 10/04: including control of propulsion units
- B60W 60/001: {Planning or execution of driving tasks}
- G06N 3/04: Architecture, e.g. interconnection topology
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/0495: Quantised networks; Sparse networks; Compressed networks
- G06N 3/063: using electronic means
- B60W 10/20: including control of steering systems
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G06N 3/045: Combinations of networks
- G06N 3/048: Activation functions
- G06N 3/049: Temporal neural networks, e.g. delay elements, oscillating neurons or pulsed inputs
- G06N 3/105: {Shells for specifying net layout}
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- B60W: CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT

## Abstract

Apparatuses, systems, and techniques to reduce a size of neural networks. In at least one embodiment, a size of a neural network is reduced by at least removing one or more neurons of the neural network and adjusting one or more layers of the neural network to compensate for the removed one or more neurons.

## Claims

### Claim 1

A processor, comprising:
one or more circuits to reduce the size of a neural network by at least removing one or more neurons of the neural network and adjusting one or more layers of the neural network to compensate for the removed one or more neurons.

### Claim 2 (depends on 1)

The processor of claim 1, wherein adjusting one or more layers of the neural network includes propagating one or more bias terms of the removed one or more neurons to a subsequent layer.

### Claim 3 (depends on 2)

The processor of claim 2, wherein propagating one or more bias terms includes calculating one or more activation function results of one or more bias terms.

### Claim 4 (depends on 1)

The processor of claim 1, wherein removing one or more neurons includes identifying zero-weighted neurons in a representation of the neural network, and removing the zero-weighted neurons.

### Claim 5 (depends on 1)

The processor of claim 1, wherein adjusting one or more layers of the neural network includes determining that removing one or more neurons would result in an element-wise operation performed between layers having operands of different length, and inserting one or more scatter operations to generate operands of equal length for the element-wise operation.

### Claim 6 (depends on 1)

The processor of claim 1, wherein the one or more circuits are also to:
import a first representation of the neural network; and
export a second representation of the size reduced neural network.

### Claim 7 (depends on 1)

The processor of claim 1, wherein adjusting one or more layers of the neural network includes:
identifying a first type of a first layer of the neural network;
identifying a second type of a second layer of the neural network, wherein the second layer is subsequent to the first layer; and
adjusting the second layer based, at least in part, on the identified first type and second type.

### Claim 8 (depends on 7)

The processor of claim 7, wherein the first type is fully-connected.

### Claim 9

A machine-readable medium having stored thereon a set of instructions, which if performed by one or more processors, cause the one or more processors to at least:
prune a neural network by at least removing one or more neurons of the neural network and adjusting one or more layers of the neural network to compensate for the pruned one or more neurons.

### Claim 10 (depends on 9)

The machine-readable medium of claim 9, wherein adjusting one or more layers of the neural network includes propagating one or more bias terms of the removed one or more neurons to a subsequent layer.

### Claim 11 (depends on 10)

The machine-readable medium of claim 10, wherein propagating one or more bias terms includes calculating one or more activation functions of one or more bias terms.

### Claim 12 (depends on 9)

The machine-readable medium of claim 9, wherein removing one or more neurons includes identifying zero-weighted neurons in a representation of the neural network, and removing the zero-weighted neurons.

### Claim 13 (depends on 9)

The machine-readable-medium of claim 9, wherein adjusting one or more layers of the neural network includes determining that removing one or more neurons would result in an element-wise operation performed between layers having operands of different length, and inserting one or more scatter operations to generate operands of equal length for the element-wise operation.

### Claim 14 (depends on 9)

The machine-readable medium of claim 9, wherein the instructions, which if performed by the one or more processors, further cause the one or more processors to:
import a first representation of the neural network; and
export a second representation of the pruned neural network.

### Claim 15 (depends on 14)

The machine-readable medium of claim 14, wherein the first representation of the neural network is a first neural network model stored according to Open Neural Network Exchange (ONNX) format and the second representation of the pruned neural network is a second neural network model stored according to ONNX format.

### Claim 16

A method, comprising:
reducing the size of a neural network by at least removing one or more neurons of the neural network and adjusting one or more layers of the neural network to compensate for the removed one or more neurons.

### Claim 17 (depends on 16)

The method of claim 16, wherein adjusting one or more layers of the neural network includes propagating one or more bias terms of the removed one or more neurons to a subsequent layer.

### Claim 18 (depends on 17)

The method of claim 17, wherein propagating one or more bias terms includes calculating one or more activation function results of one or more bias terms and calculating one or more bias terms of the subsequent layer based, at least in part, on the calculated one or more activation function results.

### Claim 19 (depends on 16)

The method of claim 16, wherein removing one or more neurons includes identifying zero-weighted neurons in a representation of the neural network, and removing the zero-weighted neurons.

### Claim 20 (depends on 16)

The method of claim 16, wherein adjusting one or more layers of the neural network includes determining that removing one or more neurons would result in an element-wise operation performed between layers having operands of different length, and inserting one or more scatter operations to generate operands of equal length for the element-wise operation.

### Claim 21 (depends on 16)

The method of claim 16, further comprising:
importing a first representation of the neural network; and
exporting a second representation of the size reduced neural network.

### Claim 22

A system, comprising:
one or more processors to reduce the size of a neural network by at least removing one or more neurons of the neural network and adjusting one or more layers of the neural network to compensate for the removed one or more neurons; and
one or more memories to store a representation of the size reduced neural network.

### Claim 23 (depends on 22)

The system of claim 22, wherein adjusting one or more layers of the neural network includes propagating one or more bias terms of the removed one or more neurons to a subsequent layer.

### Claim 24 (depends on 23)

The system of claim 23, wherein propagating one or more bias terms includes calculating one or more activation function results of one or more bias terms.

### Claim 25 (depends on 22)

The system of claim 22, wherein removing one or more neurons includes identifying zero-weighted neurons in a representation of the neural network, and removing the zero-weighted neurons.

### Claim 26 (depends on 22)

The system of claim 22, wherein adjusting one or more layers of the neural network includes determining that removing one or more neurons would result in an element-wise operation performed between layers having operands of different length, and inserting one or more scatter operations to generate operands of equal length for the element-wise operation.

### Claim 27 (depends on 22)

The system of claim 22, wherein the one or more processors are further to:
import a representation of the neural network; and
export a second representation of the size reduced neural network.

### Claim 28

A vehicle comprising:
a computer vision system that includes one or more processors to identify one or more features of a vehicle operating environment based, at least in part, on using a first neural network generated, at least in part, by reducing the size of a second neural network by at least removing one or more neurons of the second neural network and adjusting one or more layers of the second neural network to compensate for the removed one or more neurons; and
one or more of a propulsion system and a directional control system to control one or more movements of the vehicle based, at least in part, on the identified one or more features.

### Claim 29 (depends on 28)

The vehicle of claim 28, wherein adjusting one or more layers of the neural network includes propagating one or more bias terms of the removed one or more neurons to a subsequent layer.

### Claim 30 (depends on 29)

The vehicle of claim 29, wherein propagating one or more bias terms includes calculating one or more activation function results of one or more bias terms.

### Claim 31 (depends on 28)

The vehicle of claim 28, wherein removing one or more neurons includes identifying zero-weighted neurons in a representation of the neural network, and removing the zero-weighted neurons.

### Claim 32 (depends on 28)

The vehicle of claim 28, wherein adjusting one or more layers of the neural network includes determining that removing one or more neurons would result in an element-wise operation performed between layers having operands of different length, and inserting one or more scatter operations to generate operands of equal length for the element-wise operation.

### Claim 33 (depends on 28)

The vehicle of claim 28, wherein adjusting one or more layers of the neural network includes:
identifying a first type of a first layer of the neural network;
identifying a second type of a second layer of the neural network, wherein the second layer is subsequent to the first layer; and
adjusting the layer based, at least in part, on the identified first type and second type.

### Claim 34 (depends on 33)

The vehicle of claim 33, wherein the first type is convolutional.
