Compression of deep neural networks with proper use of mask
AssigneeXilinx Technology Beijing Limited
InventorsShijie Sun, Song Han, Xin Li, Yi Shan
Life of this patent
Estimating the term…
Abstract
The present invention relates to artificial neural networks, for example, deep neural networks. In particular, the present invention relates to a compression method for deep neural networks with proper use of mask and the device thereof. More specifically, the present invention relates to how to compress dense neural networks into sparse neural networks while maintaining or even improving the accuracy of the neural networks after compression.
Claims9
A method for compressing a neural network nnet0 with a mask matrix, said neural network is characterized by a plurality of matrices, said method comprising:
- mask matrix obtaining step of obtaining a mask matrix M, wherein said mask matrix M records the distribution of non-zero elements in a plurality of sparse matrices of the neural network;
- first pruning step of pruning the neural network nnet0 using said mask matrix M to obtain a pruned neural network nneti1;
- first retraining step of retraining the pruned neural network nneti1 that has a shape of the mask matrix without a mask, wherein the pruned neural network nneti1 is retrained without using said mask matrix M to obtain a dense neural network nneto1;
- second pruning step of pruning said dense neural network nneto1 using said mask matrix M to obtain a sparse neural network nneti2;
- second retraining step of retraining said sparse neural network nneti2 using said mask matrix M to obtain a final neural network nneto2.
Dependencies are inferred from the claim text.