# US 2019/0164538 A1 · Memory compression in a deep neural network

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2019/0164538 A1
- Document ID: US20190164538A1
- Status: Published application
- Kind: A1
- Application number: 16/321,097
- Filed: 2017-07-27
- Published: 2019-05-30
- Earliest claimed: 2016-07-29
- Assignees: Arizona Board of Regents on behalf of Arizona State University
- Applicants: Arizona Board of Regents on behalf of Arizona State University
- Inventors: Jae-sun Seo; Deepak Kadetotad; Sairam Arunachalam; Chaitali Chakrabarti
- Web page: https://patentgrep.com/patent/US20190164538A1

## CPC

- G06N 3/063: using electronic means
- G06F 18/24: Classification techniques
- G06N 3/0495: Quantised networks; Sparse networks; Compressed networks
- G06N 3/0499: Feedforward networks
- G06N 3/082: modifying the architecture, e.g. adding, deleting or silencing nodes or connections
- G06N 3/09: Supervised learning
- G10L 15/16: using artificial neural networks
- G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue
- G10L 15/285: {Memory allocation or algorithm optimisation to reduce hardware requirements}
- G10L 2015/223: {Execution procedure of a spoken command}
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING

## Abstract

Aspects disclosed in the detailed description include memory compression in a deep neural network (DNN). To support a DNN application, a fully connected weight matrix associated with a hidden layer(s) of the DNN is divided into a plurality of weight blocks to generate a weight block matrix with a first number of rows and a second number of columns. A selected number of weight blocks are randomly designated as active weight blocks in each of the first number of rows and updated exclusively during DNN training. The weight block matrix is compressed to generate a sparsified weight block matrix including exclusively active weight blocks. The second number of columns is compressed to reduce memory footprint and computation power, while the first number of rows is retained to maintain accuracy of the DNN, thus providing the DNN in an efficient hardware implementation without sacrificing accuracy of the DNN application.

## Claims

### Claim 1

A method for reducing memory requirement of a deep neural network (DNN) comprising:
dividing a fully connected weight matrix associated with at least one hidden layer of the DNN into a plurality of weight blocks of a determined block size to generate a weight block matrix having a first number of rows and a second number of columns;
randomly designating a selected number of active weight blocks in each of the first number of rows based on a target sparsity ratio;
updating exclusively the selected number of active weight blocks in each of the first number of rows during a DNN training for a DNN application;
compressing the weight block matrix to generate a sparsified weight block matrix comprising exclusively the selected number of active weight blocks in each of the first number of rows during the DNN training; and
storing the sparsified weight block matrix for the DNN application upon completion of the DNN training, wherein the sparsified weight block matrix is configured to be used directly for DNN classification.

### Claim 2 (depends on 1)

The method of claim 1 further comprising reducing precision of each weight in each of the selected number of active weight blocks from a floating-point precision to a fixed-point precision for the DNN classification.

### Claim 3 (depends on 1)

The method of claim 1 further comprising:
generating a binary connection coefficient identifying each non-zero weight and each zero weight in each of the selected number of active weight blocks in each of the first number of rows; and
updating exclusively the non-zero weight identified by the binary connection coefficient during the DNN training.

### Claim 4 (depends on 1)

The method of claim 1 further comprising:
generating a weight block index identifying each of the selected number of active weight blocks in the sparsified weight block matrix; and
storing the weight block index with the sparsified weight block matrix for the DNN application.

### Claim 5 (depends on 1)

The method of claim 1 further comprising performing at least one hierarchical coarse-grain sparsification (HCGS) iteration based on the sparsified weight block matrix of the DNN.

### Claim 6 (depends on 5)

The method of claim 5 further comprising:
for each active weight block in the sparsified weight block matrix:
dividing the active weight block into a plurality of HCGS weight blocks of a determined HCGS block size smaller than the determined block size; and
randomly designating a selected number of active HCGS weight blocks based on an HCGS target sparsity ratio;
compressing the sparsified weight block matrix to generate a hierarchical sparsified weight block matrix comprising exclusively the selected number of active HCGS weight blocks in each active weight block in the sparsified weight block matrix;
generating an HCGS weight block index identifying the selected number of active HCGS weight blocks in each active weight block in the sparsified weight block matrix; and
storing the HCGS weight block index for the DNN application.

### Claim 7 (depends on 1)

The method of claim 1 further comprising:
supporting a keyword detection task based on the DNN comprising an input layer, an output layer, and two hidden layers each having five hundred twelve (512) neurons;
dividing the fully connected weight matrix into sixty-four (64) 64-by-64 (64×64) weight blocks to generate the weight block matrix having eight rows and eight columns;
randomly designating two active 64×64 weight blocks in each of the eight rows;
updating the two active 64×64 weight blocks in each of the eight rows of the weight block matrix during the DNN training; and
compressing the weight block matrix to generate the sparsified weight block matrix comprising exclusively the two active 64×64 weight blocks in each of the eight rows of the weight block matrix.

### Claim 8 (depends on 1)

The method of claim 1 further comprising:
supporting a speech recognition task based on the DNN comprising an input layer, an output layer, and four hidden layers each having one thousand twenty-four (1024) neurons;
dividing the fully connected weight matrix into two hundred fifty-six (256) sixty-four-by-sixty-four (64×64) weight blocks to generate the weight block matrix having sixteen rows and sixteen columns;
randomly designating four active 64×64 weight blocks in each of the sixteen rows;
updating the four active 64×64 weight blocks in each of the sixteen rows during the DNN training; and
compressing the weight block matrix to generate the sparsified weight block matrix comprising exclusively the four active 64×64 weight blocks in each of the sixteen rows of the weight block matrix.

### Claim 9 (depends on 8)

The method of claim 8 further comprising:
for each active 64×64 weight block in the sparsified weight block matrix:
dividing the active 64×64 weight block into sixteen (16) 16-by-16 (16×16) weight blocks and organizing the 16 16×16 weight blocks into four rows and four columns; and
randomly designating two active 16×16 weight blocks in each of the four rows; and
compressing the sparsified weight block matrix to generate a hierarchical sparsified weight block matrix comprising exclusively the active 16×16 weight blocks.

### Claim 10

A non-transitory computer-readable medium comprising software with instructions configured to:
divide a fully connected weight matrix associated with at least one hidden layer of a deep neural network (DNN) into a plurality of weight blocks of a determined block size to generate a weight block matrix having a first number of rows and a second number of columns;
randomly designate a selected number of active weight blocks in each of the first number of rows based on a target sparsity ratio;
update exclusively the selected number of active weight blocks in each of the first number of rows during a DNN training for a DNN application;
compress the weight block matrix to generate a sparsified weight block matrix comprising exclusively the selected number of active weight blocks in each of the first number of rows during the DNN training; and
store the sparsified weight block matrix for the DNN application upon completion of the DNN training, wherein the sparsified weight block matrix is configured to be used directly for DNN classification.

### Claim 11 (depends on 10)

The non-transitory computer-readable medium of claim 10 wherein the software with instructions is further configured to reduce precision of each weight in each of the selected number of active weight blocks from a floating-point precision to a fixed-point precision for the DNN classification.

### Claim 12 (depends on 10)

The non-transitory computer-readable medium of claim 10 wherein the software with instructions is further configured to:
generate a binary connection coefficient identifying each non-zero weight and each zero weight in each of the selected number of active weight blocks in each of the first number of rows; and
update exclusively the non-zero weight identified by the binary connection coefficient during the DNN training.

### Claim 13 (depends on 10)

The non-transitory computer-readable medium of claim 10 wherein the software with instructions is further configured to:
generate a weight block index identifying each of the selected number of active weight blocks in the sparsified weight block matrix; and
store the weight block index with the sparsified weight block matrix for the DNN application.

### Claim 14 (depends on 10)

The non-transitory computer-readable medium of claim 10 wherein the software with instructions is further configured to perform at least one hierarchical coarse-grain sparsification (HCGS) iteration based on the sparsified weight block matrix.

### Claim 15 (depends on 14)

The non-transitory computer-readable medium of claim 14 wherein the software with instructions is further configured to:
for each active weight block in the sparsified weight block matrix:
divide the active weight block into a plurality of HCGS weight blocks of a determined HCGS block size smaller than the determined block size; and
randomly designate a selected number of active HCGS weight blocks based on an HCGS target sparsity ratio;
compress the sparsified weight block matrix to generate a hierarchical sparsified weight block matrix comprising exclusively the selected number of active HCGS weight blocks in each active weight block in the sparsified weight block matrix;
generate an HCGS weight block index identifying the selected number of active HCGS weight blocks in each active weight block in the sparsified weight block matrix; and
store the HCGS weight block index for the DNN application.

### Claim 16 (depends on 10)

The non-transitory computer-readable medium of claim 10 wherein the software with instructions is further configured to:
support a keyword detection task based on the DNN comprising an input layer, an output layer, and two hidden layers each having five hundred twelve (512) neurons;
divide the fully connected weight matrix into sixty-four (64) 64-by-64 (64×64) weight blocks to generate the weight block matrix having eight rows and eight columns;
randomly designate two active 64×64 weight blocks in each of the eight rows;
update the two active 64×64 weight blocks in each of the eight rows of the weight block matrix during the DNN training; and
compress the weight block matrix to generate the sparsified weight block matrix comprising exclusively the two active 64×64 weight blocks in each of the eight rows of the weight block matrix.

### Claim 17 (depends on 10)

The non-transitory computer-readable medium of claim 10 wherein the software with instructions is further configured to:
support a speech recognition task based on the DNN comprising an input layer, an output layer, and four hidden layers each having one thousand twenty-four (1024) neurons;
divide the fully connected weight matrix into two hundred fifty-six (256) sixty-four-by-sixty-four (64×64) weight blocks to generate the weight block matrix having sixteen rows and sixteen columns;
randomly designate four active 64×64 weight blocks in each of the sixteen rows;
update the four active 64×64 weight blocks in each of the sixteen rows during the DNN training; and
compress the weight block matrix to generate the sparsified weight block matrix comprising exclusively the four active 64×64 weight blocks in each of the sixteen rows of the weight block matrix.

### Claim 18 (depends on 17)

The non-transitory computer-readable medium of claim 17 wherein the software with instructions is further configured to:
for each active 64×64 weight block in the sparsified weight block matrix:
divide the active 64×64 weight block into sixteen (16) 16-by-16 (16×16) weight blocks and organize the 16 16×16 weight blocks into four rows and four columns; and
randomly designate two active 16×16 weight blocks in each of the four rows; and
compress the sparsified weight block matrix to generate a hierarchical sparsified weight block matrix comprising exclusively the active 16×16 weight blocks.

### Claim 19

An electronic device configured to implement a deep neural network (DNN) comprising:
one or more memory banks each configured to store a sparsified weight block matrix generated by a method comprising:
dividing a fully connected weight matrix associated with at least one hidden layer of the DNN into a plurality of weight blocks of a determined block size to generate a weight block matrix having a first number of rows and a second number of columns;
randomly designating a selected number of active weight blocks in each of the first number of rows based on a target sparsity ratio;
updating exclusively the selected number of active weight blocks in each of the first number of rows during a DNN training for a DNN application;
compressing the weight block matrix to generate the sparsified weight block matrix comprising exclusively the selected number of active weight blocks in each of the first number of rows during the DNN training; and
storing the sparsified weight block matrix in a memory bank among the one or more memory banks for the DNN application upon completion of the DNN training, wherein the sparsified weight block matrix is configured to be used directly for DNN classification; and
DNN computation circuitry coupled to the one or more memory banks and configured to:
receive a plurality of neuron inputs from a hidden layer in the DNN;
retrieve a selected sparsified weight block matrix corresponding to the hidden layer from a selected memory bank among the one or more memory banks; and
generate a neuron output based on the plurality of neuron inputs and the selected sparsified weight block matrix.

### Claim 20 (depends on 19)

The electronic device of claim 19 further comprising a controller configured to:
determine the selected sparsified weight block matrix corresponding to the hidden layer based on a finite state machine (FSM); and
provide a selection signal indicating the selected memory bank storing the selected sparsified weight block matrix to the DNN computation circuitry.