# US 12,008,474 B2 · Automatic thresholds for neural network pruning and retraining > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,008,474 B2 - Document ID: US12008474B2 - Status: Granted - Kind: B2 - Application number: 17/016,363 - Filed: 2020-09-09 - Published: 2020-12-31 - Granted: 2024-06-11 - Earliest claimed: 2017-02-10 - Assignees: Samsung Electronics Co., Ltd. - Applicants: Samsung Electronics Co., Ltd. - Inventors: Zhengping Ji; John Wakefield Brothers; Ilia Ovsiannikov; Eunsoo Shim - Web page: https://patentgrep.com/patent/US12008474B2 ## CPC - G06N 3/082: modifying the architecture, e.g. adding, deleting or silencing nodes or connections - G06N 3/045: Combinations of networks - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/0495: Quantised networks; Sparse networks; Compressed networks - G06N 3/09: Supervised learning - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS ## Abstract An embodiment includes a method, comprising: pruning a layer of a neural network having multiple layers using a threshold; and repeating the pruning of the layer of the neural network using a different threshold until a pruning error of the pruned layer reaches a pruning error allowance. ## Claims ### Claim 1 A method, comprising: pruning, by a processor, a layer of a neural network using a first threshold, the neural network comprising multiple layers, the pruning comprising configuring weights of the layer based on the first threshold; determining, by the processor, a pruning error of the layer based on a percentage of the configured weights of the layer with respect to an initial number of weights of the layer; and repeatedly configuring, by the processor, the weights of the layer until the pruning error of the layer equals a pruning error allowance for the layer, wherein the repeatedly configuring the weights of the layer allows a percentage of the number of weights remaining after the repeatedly configuring the weights of the layer with respect to the initial number of weights of the layer to be determined, and wherein each iteration of the repeatedly configuring the weights of the layer uses a different first threshold based on the pruning error. ### Claim 2 (depends on 1) The method of claim 1, wherein the pruning error is further based on a quantity of a length of a vector of the configured weights of the layer for a current iteration minus a vector of the configured weights of the layer for a previous iteration divided by a quantity of a total length of a vector of initial weights of the layer minus a length of the vector of the configured weights of the layer for the current iteration. ### Claim 3 (depends on 2) The method of claim 2, further comprising for each layer of the neural network: selecting, by the processor, the pruning error allowance for the layer; selecting, by the processor, the first threshold for the layer; and until a length of a vector of remaining non-zero weights to the total length of the vector of initial weights for the layer is within a range of a predetermined percentage for the layer, repeatedly: pruning, by the processor, the layer based on the first threshold selected for the layer, determining, by the processor, the pruning error for the layer, and changing, by the processor, the first threshold for the layer based the pruning error being less than the pruning error allowance for the layer until the pruning error is within a predetermined range of the pruning error allowance for the layer; determining, by the processor, a percentage of the length of the vector of remaining non-zero weights to the total length of the vector of initial weights for the layer; and changing, by the processor, the pruning error allowance for the layer until the percentage of the length of the vector of remaining non-zero weights to the total length of the vector of initial weights for the layer is within the range of the predetermined percentage for the layer. ### Claim 4 (depends on 3) The method of claim 3, wherein different types of layers of the neural network have different ranges for a percentage of the length of a vector of remaining non-zero weights to the total length of the vector of initial weights for the layer. ### Claim 5 (depends on 1) The method of claim 1, further comprising repeatedly pruning the layer, by the processor, using a different pruning error allowance for at least one iteration until a percentage of pruned weights to initial weights for the layer is within a range of a predetermined percentage for the layer. ### Claim 6 (depends on 1) The method of claim 1, wherein pruning the layer comprises setting, by the processor, a weight to zero based on a magnitude of the weight being less than the first threshold. ### Claim 7 (depends on 1) The method of claim 1, wherein pruning the layer comprises setting, by the processor, a weight to zero based on a magnitude of the weight being less than the first threshold scaled by a predetermined scale factor. ### Claim 8 (depends on 7) The method of claim 7, wherein the predetermined scale factor comprises a standard deviation of weights of the layer. ### Claim 9 (depends on 1) The method of claim 1, further comprising generating, by the processor, the different first threshold by increasing the first threshold based on the pruning error being less than the pruning error allowance and decreasing the first threshold based on the pruning error being greater than the pruning error allowance. ### Claim 10 (depends on 1) The method of claim 1, further comprising iteratively pruning and retraining the neural network using the first threshold after the pruning error of a currently pruned layer equals the pruning error allowance. ### Claim 11 A method, comprising: pruning a layer of a neural network, by a processor, one or more times using a different first threshold for each iteration of pruning, the pruning comprising configuring weights of the layer based on the different first threshold until a pruning error of the equals a pruning error allowance for the layer, the neural network comprising multiple layers, the pruning error of being based on a percentage of the configured weights of the layer with respect to an initial number of weights of the layer, and wherein the repeatedly configuring the weights of the layer allows a percentage of the number of weights remaining after the configuring the weights of the layer with respect to the initial number of weights of the layer to be determined; and retraining, by the processor, the neural network using only weights remaining after one or more prunings. ### Claim 12 (depends on 11) The method of claim 11, wherein the pruning error is based on a quantity of a length of a vector of the configured weights of the layer for a current iteration minus a vector of the configured weights of the layer for a previous iteration divided by a quantity of a total length of a vector of initial weight of the layer minus a length of the vector of the configured weights of the layer for the current iteration. ### Claim 13 (depends on 12) The method of claim 12, wherein the first threshold used for each iteration of pruning a first-type layer of the neural network is based on a predetermined percentage of non-zero weights to initial weights for the layer. ### Claim 14 (depends on 13) The method of claim 13, wherein the first threshold used for each iteration of pruning a second-type layer of the neural network comprises a fixed value threshold, the second-type layer being different from the first-type layer. ### Claim 15 (depends on 14) The method of claim 14, further comprising retraining, by the processor, the neural network using weights remaining after pruning first-type layers of the neural network and weights remaining after pruning second-type layers of the neural network. ### Claim 16 (depends on 15) The method of claim 15, further comprising: fixing, by the processor, weights of first type layers of the neural network; and repeatedly pruning, by the processor, second type layers of the neural network, and retraining the neural network using weights remaining after pruning second type layers. ### Claim 17 (depends on 11) The method of claim 11, further comprising adjusting, by the processor, a dropout rate for the retraining in response to a pruning rate of the pruning. ### Claim 18 A system, comprising: a memory; and a processor coupled to the memory, the processor configured to: prune a layer of a neural network using a first threshold, the neural network comprising multiple layers, the pruning comprising configuring weights of the layer based on the first threshold; determine a pruning error of the layer that is based on a percentage of the configured weights of the layer with respect to an initial number of weights of the layer; and repeatedly configure the weights of the layer until the pruning error of the layer equals a pruning error allowance for the layer, wherein the repeatedly configuring the weights of the layer allows a percentage of the number of weights remaining after the repeatedly configuring the weights of the layer with respect to the initial number of weights of the layer to be determined, and wherein each iteration of the repeatedly configuring the weights of the layer uses a different first threshold based on the pruning error. ### Claim 19 (depends on 18) The system of claim 18, wherein the pruning error is further based on a quantity of a length of a vector of the configured weights of the layer for a current iteration minus a vector of the configured weights of the layer for a previous iteration divided by a quantity of a total length of a vector of initial weights of the layer minus a length of the vector of the configured weights of the layer for the current iteration. ### Claim 20 (depends on 19) The system of claim 19, wherein for each layer of the neural network, the processor is further configured to: select the pruning error allowance for the layer; select the first threshold for the layer; and until a length of a vector of remaining non-zero weights to the total length of the vector of initial weights for the layer is within a range of a predetermined percentage for the layer, the processor is configured to repeatedly: prune the layer based on the first threshold selected for the layer, determine the pruning error for the layer, and change the first threshold for the layer based the pruning error being less than the pruning error allowance for the layer until the pruning error is within a predetermined range of the pruning error allowance for the layer; and determine a percentage of the length of the vector of remaining non-zero weights to the total length of the vector of initial weights for the layer; and change the pruning error allowance for the layer until the percentage of the length of the vector of remaining non-zero weights to the total length of the vector of initial weights for the layer is within the range of the predetermined percentage for the layer.