Compressed recurrent neural network models

A method for compressing a recurrent neural network (RNN),

• wherein the RNN is implemented by one or more computers and comprises a plurality of recurrent layers,
• wherein each of the plurality of recurrent layers of the RNN is configured to, for each of a plurality of time steps, receive a respective layer input for the time step and process the layer input to generate a respective layer output for the time step, and
• wherein each recurrent layer has a respective recurrent weight matrix, Wh, and a respective inter-layer weight matrix, Wx, the method comprising, for one of the plurality of recurrent layers:
• generating a first compressed weight matrix, Zhl, and a projection matrix, Pl, wherein the product of the first compressed weight matrix and the projection matrix approximates the recurrent weight matrix of the one of the plurality of recurrent layers;
• and
• generating a second compressed weight matrix, Zxl, based on the first compressed weight matrix, Zhl, and the projection matrix, Pl, wherein the product of the second compressed weight matrix and the projection matrix approximates the inter-layer weight matrix of the one of the plurality of recurrent layers.

A system comprising one or more computers and one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to perform the operations for compressing a recurrent neural network (RNN),

• wherein the RNN is implemented by one or more computers and comprises a plurality of recurrent layers,
• wherein each of the plurality of recurrent layers of the RNN is configured to, for each of a plurality of time steps, receive a respective layer input for the time step and process the layer input to generate a respective layer output for the time step, and
• wherein each recurrent layer has a respective recurrent weight matrix, Wh, and a respective inter-layer weight matrix, Wx, the operations comprising, for one of the plurality of recurrent layers:
• generating a first compressed weight matrix, Zhl, and a projection matrix, Pl, wherein the product of the first compressed weight matrix and the projection matrix approximates the recurrent weight matrix of the one of the plurality of recurrent layers;
• and
• generating a second compressed weight matrix, Zxl, based on the first compressed weight matrix, Zhl, and the projection matrix, Pl, wherein the product of the second compressed weight matrix and the projection matrix approximates the inter-layer weight matrix of the one of the plurality of recurrent layers.