# US 12,602,580 B2 · Dataset dependent low rank decomposition of neural networks > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,602,580 B2 - Document ID: US12602580B2 - Status: Granted - Kind: B2 - Application number: 16/453,380 - Filed: 2019-06-26 - Published: 2020-12-31 - Granted: 2026-04-14 - Earliest claimed: 2019-06-26 - Assignees: International Business Machines Corporation - Applicants: International Business Machines Corporation - Inventors: Anamitra R. Choudhury; Saurabh Goyal; Vivek Sharma; Venkatesan T. Chakaravarthy; Yogish Sabharwal; Ashish Verma - Web page: https://patentgrep.com/patent/US12602580B2 ## CPC - G06N 3/045: Combinations of networks - G06N 3/08: Learning methods - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/0495: Quantised networks; Sparse networks; Compressed networks - G06N 3/082: modifying the architecture, e.g. adding, deleting or silencing nodes or connections - G06N 20/00: Machine learning - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS ## Abstract Methods, systems, and computer program products for dataset dependent low rank decomposition of neural networks are provided herein. A computer-implemented method includes obtaining a target dataset and a trained model of a neural network; providing at least a portion of the target dataset to the trained model; determining relevance of each of one or more of filters of the neural network and channels of the neural network to the target dataset based on the provided portion, wherein the one or more of the filters and the channels correspond to at least one layer of the neural network; and compressing the trained model of the neural network based at least in part on the determined relevancies. ## Claims ### Claim 1 A computer-implemented method, the method comprising: providing a portion of a first dataset to a trained model of a neural network trained on a second dataset that is different than the first dataset, wherein the first dataset comprises a plurality of training samples for a set of classes, and wherein the portion of the first dataset comprises at least one of: (i) a designated number of training samples from two or more classes in the set of classes; and (ii) all training samples in a subset of the set of classes; determining a non-binary relevance value for each of one or more of filters of the neural network and channels of the neural network to the first dataset based on the provided portion, wherein the one or more of the filters and the channels correspond to at least one layer of the neural network, and wherein the determining is based on statistics computed for a particular number of epochs of training using the portion of the first dataset; adjusting at least one first weight tensor corresponding to the at least one layer of the neural network, wherein the at least one first weight tensor comprises weights for the one or more of the filters and the channels of the at least one layer, and wherein the adjusting comprises element-wise scaling of the weights in the at least one first weight tensor by the determined non-binary relevance value for each of the one or more of the filters and the channels; generating at least one second weight tensor by performing a weighted low rank factorization process that approximates the scaled weights of the adjusted at least one first weight tensor, wherein a rank of the at least one second weight tensor is smaller than a rank of the at least one first weight tensor, and wherein the weighted low rank factorization process comprises reducing a weighted error measure corresponding to a total deviation across corresponding tensor elements of the adjusted at least one first weight tensor and tensor elements of the at least one second weight tensor; using the at least one second weight tensor to compress the trained model of the neural network by replacing, for the at least one layer, the at least one first weight tensor with the at least one second weight tensor; and retraining the compressed trained model with the first dataset; wherein the method is carried out by at least one computing device. ### Claim 2 (depends on 1) The computer-implemented method of claim 1, wherein the determining comprises: performing the particular number of epochs of the training using the portion of the first dataset; and computing the statistics based on activations of the one or more of the filters and the channels during the training. ### Claim 3 (depends on 1) The computer-implemented method of claim 1, wherein the first dataset comprises a plurality of images, and wherein the determining comprises: determining which of the one or more of the filters and the channels are activated for one or more of: all of the plurality of images of the first dataset; images of a particular class of interest; images of a class with a number of training samples that is below a threshold; and a particular layer of the neural network. ### Claim 4 (depends on 1) The computer-implemented method of claim 1, wherein the rank of the at least one second weight tensor is determined using a higher order singular value decomposition process. ### Claim 5 (depends on 1) The computer-implemented method of claim 1, wherein the tensor elements of the at least one second weight tensor depend on the determined non-binary relevance value for each of the one or more of the filters and the channels such that one or more of the filters and the channels that are determined to be more relevant to the first dataset are retained in the at least one second weight tensor. ### Claim 6 (depends on 1) The computer-implemented method of claim 1, comprising: configuring the rank of the at least one second weight tensor based at least in part on one or more constraints corresponding to one or more of processing requirements and memory requirements of the compressed trained model. ### Claim 7 (depends on 1) The computer-implemented method of claim 1, wherein the reducing of the weighted error measure comprises applying one or more of a gradient descent process and a conjugate gradient process, and wherein the weighted error measure comprises a matrix norm. ### Claim 8 (depends on 1) The computer-implemented method of claim 1, wherein the trained model comprises a compressed model generated based on one or more of: a weight-based pruning process; at least one of a quantization and a weight sharing process; a relative indexing process; and an encoding process, and wherein the trained model is further compressed using the at least one second weight tensor. ### Claim 9 (depends on 1) The computer-implemented method of claim 1, comprising: outputting the trained model, from the at least one computing device to at least one separate computing device, wherein the at least one separate computing device has one or more of limited processing capabilities and memory capabilities relative to the at least one computing device. ### Claim 10 (depends on 1) The computer-implemented method of claim 1, wherein the rank of the at least one second weight tensor is configured to control an amount of compression applied to the trained model. ### Claim 11 (depends on 2) The computer-implemented method of claim 2, wherein a particular non-binary relevance value of a given one of the one or more of the filters is determined by performing one or more of a step function, a sigmoid function, and a logistic function based on the statistics. ### Claim 12 (depends on 1) The computer-implemented method of claim 1, wherein the generated at least one second weight tensor approximates all of the scaled weights of the adjusted at least one first weight tensor. ### Claim 13 (depends on 1) The computer-implemented method of claim 1, wherein the weighted low rank factorization process comprises a Tucker Decomposition process, and wherein the weighted error measure comprises a Frobenius norm. ### Claim 14 A computer program product comprising a non-transitory computer readable storage medium having program instructions embodied therewith, the program instructions executable by a computing device to cause the computing device to: provide a portion of a first dataset to a trained model of a neural network trained on a second dataset that is different than the first dataset, wherein the first dataset comprises a plurality of training samples for a set of classes, and wherein the portion of the first dataset comprises at least one of: (i) a designated number of training samples from two or more classes in the set of classes; and (ii) all training samples in a subset of the set of classes; determine a non-binary relevance value for each of one or more of filters of the neural network and channels of the neural network to the first dataset based on the provided portion, wherein the one or more of the filters and the channels correspond to at least one layer of the neural network, and wherein the determining is based on statistics computed for a particular number of epochs of training using the portion of the first dataset; adjust at least one first weight tensor corresponding to the at least one layer of the neural network, wherein the at least one first weight tensor comprises weights for the one or more of the filters and the channels of the at least one layer, and wherein the adjusting comprises element-wise scaling of the weights in the at least one first weight tensor by the determined non-binary relevance value for each of the one or more of the filters and the channels; generate at least one second weight tensor by performing a weighted low rank factorization process that approximates the scaled weights of the adjusted at least one first weight tensor, wherein a rank of the at least one second weight tensor is smaller than a rank of the at least one first weight tensor, and wherein the weighted low rank factorization process comprises reducing a weighted error measure corresponding to a total deviation across corresponding tensor elements of the adjusted at least one first weight tensor and tensor elements of the at least one second weight tensor; use the at least one second weight tensor to compress the trained model of the neural network by replacing, for the at least one layer, the at least one first weight tensor with the at least one second weight tensor; and retrain the compressed trained model with the first dataset. ### Claim 15 (depends on 14) The computer program product of claim 14, wherein the determining comprises: performing the particular number of epochs of the training using the portion of the first dataset; and computing the statistics based on activations of the one or more of the filters and the channels during the training. ### Claim 16 (depends on 14) The computer program product of claim 14, wherein the first dataset comprises a plurality of images, and wherein the determining comprises: determining which of the one or more of the filters and the channels are activated for one or more of: all of the plurality of images of the first dataset; images of a particular class of interest; images of a class with a number of training samples that is below a threshold; and a particular layer of the neural network. ### Claim 17 (depends on 14) The computer program product of claim 14, wherein the weighted low rank factorization process comprises a Tucker Decomposition process, and wherein the weighted error measure comprises a Frobenius norm. ### Claim 18 A system comprising: a memory; and at least one processor operably coupled to the memory and configured to: provide a portion of a first dataset to a trained model of a neural network trained on a second dataset that is different than the first dataset, wherein the first dataset comprises a plurality of training samples for a set of classes, and wherein the portion of the first dataset comprises at least one of: (i) a designated number of training samples from two or more classes in the set of classes; and (ii) all training samples in a subset of the set of classes; determine a non-binary relevance value for each of one or more of filters of the neural network and channels of the neural network to the first dataset based on the provided portion, wherein the one or more of the filters and the channels correspond to at least one layer of the neural network, and wherein the determining is based on statistics computed for a particular number of epochs of training using the portion of the first dataset; adjust at least one first weight tensor corresponding to the at least one layer of the neural network, wherein the at least one first weight tensor comprises weights for the one or more of the filters and the channels of the at least one layer, and wherein the adjusting comprises element-wise scaling of the weights in the at least one first weight tensor by the determined non-binary relevance value for each of the one or more of the filters and the channels; generate at least one second weight tensor by performing a weighted low rank factorization process that approximates the scaled weights of the adjusted at least one first weight tensor, wherein a rank of the at least one second weight tensor is smaller than a rank of the at least one first weight tensor, and wherein the weighted low rank factorization process comprises reducing a weighted error measure corresponding to a total deviation across corresponding tensor elements of the adjusted at least one first weight tensor and tensor elements of the at least one second weight tensor; use the at least one second weight tensor to compress the trained model of the neural network by replacing, for the at least one layer, the at least one first weight tensor with the at least one second weight tensor; and retrain the compressed trained model with the first dataset. ### Claim 19 (depends on 18) The system of claim 18, wherein the weighted low rank factorization process comprises a Tucker Decomposition process, and wherein the weighted error measure comprises a Frobenius norm. ### Claim 20 A computer-implemented method, the method comprising: providing a portion of a first dataset to a trained model of a convolutional neural network trained on a second dataset that is different than the first dataset, wherein the first dataset comprises a plurality of training samples for a set of classes, and wherein the portion of the first dataset comprises at least one of: (i) a designated number of training samples from two or more classes in the set of classes; and (ii) all training samples in a subset of the set of classes; determining inherent redundancies of the trained model and a non-binary relevance value for each of one or more of filters of the convolutional neural network and channels of the convolutional neural network to the first dataset based on the provided portion, wherein the one or more of the filters and the channels correspond to at least one layer of the convolutional neural network, and wherein the determining is based on statistics computed for a particular number of epochs of training using the portion of the first dataset; adjusting at least one first weight tensor corresponding to the at least one layer of the convolutional neural network based at least in part on the determined inherent redundancies, wherein the at least one first weight tensor comprises weights for the one or more of the filters and the channels of the at least one layer, and wherein the adjusting comprises element-wise scaling of the weights in the at least one first weight tensor by the determined non-binary relevance value for each of the one or more of the filters and the channels; generating at least one second weight tensor by performing a weighted low rank factorization process that approximates the scaled weights of the adjusted at least one first weight tensor, wherein a rank of the at least one second weight tensor is smaller than a rank of the at least one first weight tensor, and wherein the weighted low rank factorization process comprises reducing a weighted error measure corresponding to a total deviation across corresponding tensor elements of the adjusted at least one first weight tensor and tensor elements of the at least one second weight tensor; using the at least one second weight tensor to compress the trained model of the convolutional neural network by replacing, for the at least one layer, the at least one first weight tensor with the at least one second weight tensor; and retraining the compressed trained model with the first dataset; wherein the method is carried out by at least one computing device.