# US 2021/0089916 A1 · Compressed recurrent neural network models

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0089916 A1
- Document ID: US20210089916A1
- Status: Published application
- Kind: A1
- Application number: 17/112,966
- Filed: 2020-12-04
- Published: 2021-03-25
- Earliest claimed: 2016-02-03
- Assignees: Google Inc.
- Applicants: Google LLC
- Inventors: Ouais Alsharif; Rohit Prakash Prabhavalkar; Ian C. McGraw; Antoine Jean Bruguier
- Web page: https://patentgrep.com/patent/US20210089916A1

## CPC

- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G06N 3/0495: Quantised networks; Sparse networks; Compressed networks
- G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU]
- G06N 3/049: Temporal neural networks, e.g. delay elements, oscillating neurons or pulsed inputs
- G06N 3/09: Supervised learning
- G06N 20/00: Machine learning
- G05B 2219/33025: Recurrent artificial neural network
- G05B 2219/40326: Singular value decomposition
- G06F 17/16: Matrix or vector computation {, e.g. matrix-matrix or matrix-vector multiplication, matrix factorization (matrix transposition G06F7/78)}
- G06N 3/04: Architecture, e.g. interconnection topology
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G05B: CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS (systems for controlling or regulating non-electric variables G05D; systems for regulating electric or magnetic variables G05F; control devices or systems insofar as characterised by mechanical features only G05G)
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for implementing a compressed recurrent neural network (RNN). One of the systems includes a compressed RNN, the compressed RNN comprising a plurality of recurrent layers, wherein each of the recurrent layers has a respective recurrent weight matrix and a respective inter-layer weight matrix, and wherein at least one of recurrent layers is compressed such that a respective recurrent weight matrix of the compressed layer is defined by a first compressed weight matrix and a projection matrix and a respective inter-layer weight matrix of the compressed layer is defined by a second compressed weight matrix and the projection matrix.

## Claims

### Claim 1

A computer-implemented method that when executed on data processing hardware causes the data processing hardware to perform operations comprising:
re-configuring a trained recurrent neural network (RNN) by, for at least one recurrent layer of a plurality of uncompressed recurrent layers of an uncompressed version of the trained RNN, compressing the recurrent layer by:
generating a first compressed weight matrix, Zhl, and a projection matrix, Pl, based on a respective recurrent weight matrix, Wh, for the recurrent layer of the uncompressed version of the trained RNN;
generating a second compressed weight matrix, Zxl, based on the first compressed weight matrix, Zhl, and the projection matrix, Pl; and
replacing the respective recurrent weight matrix, Wh, with the product of the first compressed weight matrix, Zhl, and the projection matrix, Pl; and
transmitting the re-configured trained RNN having the at least one compressed recurrent layer to a device, the re-configured trained RNN having the at least one compressed recurrent layer configured to receive a respective neural network input at each of multiple time steps and generate a respective neural network output at each of the multiple time steps.

### Claim 2 (depends on 1)

The computer-implemented method of claim 1, wherein respective ranks of the first compressed weight matrix, Zhl, and the projection matrix, Pl, are less than a respective rank of the respective recurrent weight matrix, Wh, for the recurrent layer.

### Claim 3 (depends on 1)

The computer-implemented method of claim 1, wherein re-configuring the trained RNN further comprises re-configuring the trained RNN by compressing the recurrent layer by replacing a respective inter-layer weight matrix, Wx, for the recurrent layer of the uncompressed version of the trained RNN with the product of the second compressed weight matrix, Zxl, and the projection matrix, Pl.

### Claim 4 (depends on 3)

The computer-implemented method of claim 3, wherein respective ranks of the second compressed weight matrix, Zxl, and the projection matrix, Pl, are less than a respective rank of the respective inter-layer weight matrix, Wx, for the recurrent layer.

### Claim 5 (depends on 1)

The computer-implemented method of claim 1, wherein generating the first compressed weight matrix, Zhl, and the projection matrix, Pl, comprises:
determining a respective singular value decomposition (SVD) of the respective recurrent weight matrix, Wh, for the recurrent layer; and
generating the first compressed weight matrix, Zhl, and the projection matrix, Pl, based on the respective SVD of the respective recurrent weight matrix, Wh, for the recurrent laver.

### Claim 6 (depends on 1)

The computer-implemented method of claim 1, wherein each recurrent layer of the plurality of uncompressed recurrent layers is configured to, for each of a plurality of time steps, generate a respective layer output for the time step by applying an inter-layer weight matrix, Wxl−1, for a previous recurrent layer to a current input to the recurrent layer and applying the respective recurrent weight matrix, Whl, for the recurrent layer to a respective recurrent input to the recurrent layer.

### Claim 7 (depends on 6)

The computer-implemented method of claim 6, further comprising, after compressing the at least one recurrent layer of the plurality of uncompressed recurrent layers, generating the respective layer output, for each time step, by applying the first compressed weight matrix, Zhl, and the projection matrix, Pl, to the respective recurrent input to the recurrent layer.

### Claim 8 (depends on 1)

The computer-implemented method of claim 1, wherein the RNN comprises an acoustic model.

### Claim 9 (depends on 1)

The computer-implemented method of claim 1, wherein the RNN comprises a speech recognition model.

### Claim 10 (depends on 1)

The computer-implemented method of claim 1, wherein, after compressing the at least one recurrent layer of the plurality of uncompressed recurrent layers, the re-configured trained RNN comprises at least a 68% compression of the uncompressed version of the trained RNN prior to compressing the at least one recurrent layer of the plurality of uncompressed recurrent layers.

### Claim 11 (depends on 1)

The computer-implemented method of claim 1, wherein, after compressing the at least one recurrent layer of the plurality of uncompressed recurrent layers, a word error rate of the re-configured trained RNN is within 5% of a word error rate of the uncompressed version of the trained RNN prior to compressing the at least one recurrent layer of the plurality of uncompressed recurrent layers.

### Claim 12

A system comprising:
data processing hardware; and
memory hardware in communication with the data processing hardware and storing instructions that when executed on the data processing hardware cause the data processing hardware to perform operations comprising:
re-configuring a trained recurrent neural network (RNN) by, for at least one recurrent layer of a plurality of uncompressed recurrent layers of an uncompressed version of the trained RNN, compressing the recurrent layer by:
generating a first compressed weight matrix, Zhl, and a projection matrix, Pl, based on a respective recurrent weight matrix, Wh, for the recurrent layer of the uncompressed version of the trained RNN;
generating a second compressed weight matrix, Zxl, based on the first compressed weight matrix, Zhl, and the projection matrix, Pl; and
replacing the respective recurrent weight matrix, Wh, with the product of the first compressed weight matrix, Zhl, and the projection matrix, Pl; and
transmitting the re-configured trained RNN having the at least one compressed recurrent layer to a device, the re-configured trained RNN having the at least one compressed recurrent layer configured to receive a respective neural network input at each of multiple time steps and generate a respective neural network output at each of the multiple time steps.

### Claim 13 (depends on 12)

The system of claim 12, wherein respective ranks of the first compressed weight matrix, Zhl, and the projection matrix, Pl, are less than a respective rank of the respective recurrent weight matrix, Wh, for the recurrent layer.

### Claim 14 (depends on 12)

The system of claim 12, wherein re-configuring the trained RNN further comprises re-configuring the trained RNN by compressing the recurrent layer by replacing a respective inter-layer weight matrix, Wx, for the recurrent layer of the uncompressed version of the trained RNN with the product of the second compressed weight matrix, Zxl, and the projection matrix, Pl.

### Claim 15 (depends on 14)

The system of claim 14, wherein respective ranks of the second compressed weight matrix, Zxl, and the projection matrix, Pl, are less than a respective rank of the respective inter-layer weight matrix, Wx, for the recurrent layer

### Claim 16 (depends on 12)

The system of claim 12, wherein generating the first compressed weight matrix, Zhl, and the projection matrix, Pl, comprises:
determining a respective singular value decomposition (SVD) of the respective recurrent weight matrix, Wh, for the recurrent layer-, and
generating the first compressed weight matrix, Zhl, and the projection matrix, Pl, based on the respective SVD of the respective recurrent weight matrix, Wh, for the recurrent layer.

### Claim 17 (depends on 12)

The system of claim 12, wherein each recurrent layer of the plurality of uncompressed recurrent layers is configured to, for each of a plurality of time steps, generate a respective layer output for the time step by applying an inter-layer weight matrix, Wxl−1, for a previous recurrent layer to a current input to the recurrent layer and applying the respective recurrent weight matrix, Whl, for the recurrent layer to a respective recurrent input to the recurrent layer.

### Claim 18 (depends on 17)

The system of claim 17, wherein the operations further comprise, after compressing the at least one recurrent layer of the plurality of uncompressed recurrent layers, generating the respective layer output, for each time step, by applying the first compressed weight matrix, Zhl, and the projection matrix, Pl, to the respective recurrent input to the recurrent layer.

### Claim 19 (depends on 12)

The system of claim 12, wherein the RNN comprises an acoustic model.

### Claim 20 (depends on 12)

The system of claim 12, wherein the RNN comprises a speech recognition model.

### Claim 21 (depends on 12)

The system of claim 12, wherein, after compressing the at least one recurrent layer of the plurality of uncompressed recurrent layers, the re-configured trained RNN comprises at least a 68% compression of the uncompressed version of the trained RNN prior to compressing the at least one recurrent layer of the plurality of uncompressed recurrent layers.

### Claim 22 (depends on 12)

The system of claim 12, wherein, after compressing the at least one recurrent layer of the plurality of uncompressed recurrent layers, a word error rate of the re-configured trained RNN is within 5% of a word error rate of the uncompressed version of the trained RNN prior to compressing the at least one recurrent layer of the plurality of uncompressed recurrent layers.
