Dataset dependent low rank decomposition of neural networks

A computer program product comprising a non-transitory computer readable storage medium having program instructions embodied therewith, the program instructions executable by a computing device to cause the computing device to:

• provide a portion of a first dataset to a trained model of a neural network trained on a second dataset that is different than the first dataset, wherein the first dataset comprises a plurality of training samples for a set of classes, and wherein the portion of the first dataset comprises at least one of:
• (i) a designated number of training samples from two or more classes in the set of classes;
• and (ii) all training samples in a subset of the set of classes;
• determine a non-binary relevance value for each of one or more of filters of the neural network and channels of the neural network to the first dataset based on the provided portion, wherein the one or more of the filters and the channels correspond to at least one layer of the neural network, and wherein the determining is based on statistics computed for a particular number of epochs of training using the portion of the first dataset;
• adjust at least one first weight tensor corresponding to the at least one layer of the neural network, wherein the at least one first weight tensor comprises weights for the one or more of the filters and the channels of the at least one layer, and wherein the adjusting comprises element-wise scaling of the weights in the at least one first weight tensor by the determined non-binary relevance value for each of the one or more of the filters and the channels;
• generate at least one second weight tensor by performing a weighted low rank factorization process that approximates the scaled weights of the adjusted at least one first weight tensor, wherein a rank of the at least one second weight tensor is smaller than a rank of the at least one first weight tensor, and wherein the weighted low rank factorization process comprises reducing a weighted error measure corresponding to a total deviation across corresponding tensor elements of the adjusted at least one first weight tensor and tensor elements of the at least one second weight tensor;
• use the at least one second weight tensor to compress the trained model of the neural network by replacing, for the at least one layer, the at least one first weight tensor with the at least one second weight tensor;
• and
• retrain the compressed trained model with the first dataset.

A system comprising:

• a memory;
• and
• at least one processor operably coupled to the memory and configured to:
• provide a portion of a first dataset to a trained model of a neural network trained on a second dataset that is different than the first dataset, wherein the first dataset comprises a plurality of training samples for a set of classes, and wherein the portion of the first dataset comprises at least one of:
• (i) a designated number of training samples from two or more classes in the set of classes;
• and (ii) all training samples in a subset of the set of classes;
• determine a non-binary relevance value for each of one or more of filters of the neural network and channels of the neural network to the first dataset based on the provided portion, wherein the one or more of the filters and the channels correspond to at least one layer of the neural network, and wherein the determining is based on statistics computed for a particular number of epochs of training using the portion of the first dataset;
• adjust at least one first weight tensor corresponding to the at least one layer of the neural network, wherein the at least one first weight tensor comprises weights for the one or more of the filters and the channels of the at least one layer, and wherein the adjusting comprises element-wise scaling of the weights in the at least one first weight tensor by the determined non-binary relevance value for each of the one or more of the filters and the channels;
• generate at least one second weight tensor by performing a weighted low rank factorization process that approximates the scaled weights of the adjusted at least one first weight tensor, wherein a rank of the at least one second weight tensor is smaller than a rank of the at least one first weight tensor, and wherein the weighted low rank factorization process comprises reducing a weighted error measure corresponding to a total deviation across corresponding tensor elements of the adjusted at least one first weight tensor and tensor elements of the at least one second weight tensor;
• use the at least one second weight tensor to compress the trained model of the neural network by replacing, for the at least one layer, the at least one first weight tensor with the at least one second weight tensor;
• and
• retrain the compressed trained model with the first dataset.

A computer-implemented method, the method comprising:

• providing a portion of a first dataset to a trained model of a convolutional neural network trained on a second dataset that is different than the first dataset, wherein the first dataset comprises a plurality of training samples for a set of classes, and wherein the portion of the first dataset comprises at least one of:
• (i) a designated number of training samples from two or more classes in the set of classes;
• and (ii) all training samples in a subset of the set of classes;
• determining inherent redundancies of the trained model and a non-binary relevance value for each of one or more of filters of the convolutional neural network and channels of the convolutional neural network to the first dataset based on the provided portion, wherein the one or more of the filters and the channels correspond to at least one layer of the convolutional neural network, and wherein the determining is based on statistics computed for a particular number of epochs of training using the portion of the first dataset;
• adjusting at least one first weight tensor corresponding to the at least one layer of the convolutional neural network based at least in part on the determined inherent redundancies, wherein the at least one first weight tensor comprises weights for the one or more of the filters and the channels of the at least one layer, and wherein the adjusting comprises element-wise scaling of the weights in the at least one first weight tensor by the determined non-binary relevance value for each of the one or more of the filters and the channels;
• generating at least one second weight tensor by performing a weighted low rank factorization process that approximates the scaled weights of the adjusted at least one first weight tensor, wherein a rank of the at least one second weight tensor is smaller than a rank of the at least one first weight tensor, and wherein the weighted low rank factorization process comprises reducing a weighted error measure corresponding to a total deviation across corresponding tensor elements of the adjusted at least one first weight tensor and tensor elements of the at least one second weight tensor;
• using the at least one second weight tensor to compress the trained model of the convolutional neural network by replacing, for the at least one layer, the at least one first weight tensor with the at least one second weight tensor;
• and
• retraining the compressed trained model with the first dataset;
• wherein the method is carried out by at least one computing device.