Neural network acceleration and embedding compression systems and methods with activation sparsification

A computer-implemented activation compressor, comprising:

• a network model module configured to compute the activation value of a plurality of neurons and to compute a prediction, the network module including a collection of variables that parameterize a Deep Neural Network (DNN), the Deep Neural Network being modularized into a plurality of layers, each layer taking the output of one or more previous layers as input data and computing a new set of outputs;
• a loss function module, communicatively coupled to the network model module, configured to compute a first penalty value that measures the discrepancy between the prediction and a ground-truth answer, the loss function module configured to determine a loss function based on a specific task with N input samples and corresponding expected outputs that measure the predictive performance of the Deep Neural Network;
• an activation regularizer, communicatively coupled to the network model module, configured to compute a second penalty value that measures the level of activation of all neurons of the network model module, the activation regularizer comprising a summation of a plurality of penalty functions, each penalty function being computed over a neuron activation matrix with a first dimension of the matrix equal to the number of samples and a second dimension of the matrix equal to the total number of neurons, the activation regularizer augmenting the loss function to measure the activation level of the neuron activation matrix formed by all neurons of the Deep Neural Network on the input samples, thereby sparsifying the number of active neurons in the Deep Neural Network when being minimized;
• a data reader module, communicatively coupled to the network model module including a pipeline for obtaining input data for the Deep Neural Network from a plurality of data sources;
• and
• an optimizer, communicatively coupled to the loss function module and to the activation regularizer, configured to adjust the values of variables in the network model module for decreasing an objective function that includes the loss function and the activation regularizer.

A computer-implemented method for efficient computation of a deep neural network (DNN), comprising:

• (a) providing a network model module for computing the activation value of a plurality of neurons and to compute a prediction, the network model module including a collection of variables that parameterize a Deep Neural Network, the Deep Neural Network being modularized into a plurality of layers, each layer taking the output of one or more previous layers as input data and computes a new set of outputs;
• (b) evaluating gradient of an objective function that includes an activation regularizer by backpropagation through the network model module and a data reader, the evaluating gradient including requesting a gradient of an activation regularizer with respect to the parameters, the activation regularizer comprising a summation of a plurality of penalty functions, each penalty function being computed over a neuron activation matrix, with a first dimension of the matrix equal to the number of samples and a second dimension of the matrix equal to the total number of neurons in the Deep Neural Network, the gradient specifying the change direction of weight values to sparsify the number of non-zero entries in the neuron activation matrix;
• (c) updating DNN parameters based on the gradient by an optimizer for adjusting the values of the collection of variables in the network model module for decreasing the objective function;
• (d) determining by the optimizer whether the change in objective or the magnitude of gradient is sufficiently small based on a predetermined threshold, to indicate that the objective has converged, the optimizer module configured to adjust the values of variables in the network model module for decreasing an objective function that includes the loss function and the activation regularizer,
• when the objective has not converged, iteratively repeating steps (b) and (c), and
• when the objective has converged, generating an output with the activation Deep Neural Network.