Back to results
US 12,743,609 B2Granted
Apparatus and a method for neural network compression
AssigneeNokia Technologies Oy
InventorsCaglar Aytekin, Francesco Cricri
Life of this patent
Estimating the term…
Abstract
There is provided an apparatus comprising means for training a neural network, wherein the training comprises applying a loss function configured to increase sparsity of a weight tensor of the neural network and to cause a plurality of non-zero elements of the weight tensor to be substantially equal to each other; and means for entropy coding the weight tensor to obtain a compressed neural network.
Claims16
An apparatus comprising:
- at least one processor;
- and
- at least one memory storing instructions that, when executed by the at least one processor, cause the apparatus at least to:
- train a neural network, wherein to train the neural network, the apparatus is further caused to apply a loss function configured to increase a sparsity of a weight tensor of the neural network;
- wherein the loss function comprises a task loss added to:
- a compression loss weight multiplied with a compression loss;
- wherein the compression loss of the loss function is defined by:
- an L1 norm of the weight tensor of the neural network divided by an L2 norm of the weight tensor of the neural network added to a transformation factor that is multiplied with a square of the L2 norm of the weight tensor of the neural network divided by the L1 norm of the weight tensor of the neural network;
- entropy encode the weight tensor of the neural network to obtain a compressed neural network, wherein the sparsity of the weight tensor of the neural network is increased following the training of the neural network by applying the loss function configured to increase the sparsity of the weight tensor of the neural network to obtain a sparse weight tensor of the neural network;
- wherein at least a portion of elements of the sparse weight tensor of the neural network are substantially equal to zero such that an absolute value of a difference between a value of any element of the portion of elements of the sparse weight tensor of the neural network and zero is less than or equal to a threshold;
- and
- transmit the compressed neural network over a communication network to at least one device.
9 more elements
A method comprising:
- training a neural network, wherein the training comprises applying a loss function configured to increase a sparsity of a weight tensor of the neural network;
- wherein the loss function comprises a task loss added to:
- a compression loss weight multiplied with a compression loss;
- wherein the compression loss of the loss function is defined by:
- an L1 norm of the weight tensor of the neural network divided by an L2 norm of the weight tensor of the neural network added to a transformation factor that is multiplied with a square of the L2 norm of the weight tensor of the neural network divided by the L1 norm of the weight tensor of the neural network;
- entropy coding the weight tensor of the neural network to obtain a compressed neural network, wherein the sparsity of the weight tensor of the neural network is increased following the training of the neural network by applying the loss function configured to increase the sparsity of the weight tensor of the neural network to obtain a sparse weight tensor of the neural network;
- wherein at least a portion of elements of the sparse weight tensor of the neural network are substantially equal to zero such that an absolute value of a difference between any element of the portion of elements of the sparse weight tensor of the neural network and zero is less than or equal to a threshold;
- and
- transmitting the compressed neural network over a communication network to at least one device.
6 more elements
A non-transitory computer readable storage medium comprising instructions stored thereon that cause an apparatus or a system to:
- train a neural network, wherein the training comprises applying a loss function configured to increase a sparsity of a weight tensor of the neural network;
- wherein the loss function comprises a task loss added to:
- a compression loss weight multiplied with a compression loss;
- wherein the compression loss of the loss function is defined by:
- an L1 norm of the weight tensor of the neural network divided by an L2 norm of the weight tensor of the neural network added to a transformation factor that is multiplied with a square of the L2 norm of the weight tensor of the neural network divided by the L1 norm of the weight tensor of the neural network;
- entropy encode the weight tensor of the neural network to obtain a compressed neural network, wherein the sparsity of the weight tensor of the neural network is increased following the training of the neural network by applying the loss function configured to increase the sparsity of the weight tensor of the neural network to obtain a sparse weight tensor of the neural network;
- wherein at least a portion of elements of the sparse weight tensor of the neural network are substantially equal to zero such that an absolute value of a difference between any element of the portion of elements of the sparse weight tensor of the neural network and zero is less than or equal to a threshold;
- and
- transmit the compressed neural network over a communication network to at least one device.
6 more elements
Dependencies are inferred from the claim text.