# US 12,002,453 B2 · Methods and devices for irregular pruning for automatic speech recognition > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,002,453 B2 - Document ID: US12002453B2 - Status: Granted - Kind: B2 - Application number: 17/213,010 - Filed: 2021-03-25 - Published: 2022-09-29 - Granted: 2024-06-04 - Earliest claimed: 2021-03-25 - Assignees: Beijing Transtreams Technology Co. Ltd. - Applicants: Kwai Inc. - Inventors: Yongxiong Ren; Bingbing Li; Yang Liu; Lingzhi Liu - Web page: https://patentgrep.com/patent/US12002453B2 ## CPC - G10L 15/16: using artificial neural networks - G06N 3/0455: Auto-encoder networks; Encoder-decoder networks - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/0495: Quantised networks; Sparse networks; Compressed networks - G06N 3/082: modifying the architecture, e.g. adding, deleting or silencing nodes or connections - G10L 15/28: Constructional details of speech recognition systems - G10L 25/30: using neural networks - G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS ## Abstract A method and an apparatus for automatic speech recognition are provided. The method includes: generating a weight matrix for a layer of a plurality of layers in a neural network; dividing the weight matrix into a plurality of blocks, each block including a plurality of weights; selecting a pre-determined percentage of weights from at least one block for block-wise pruning; and generating a block-wise pruned weight matrix by setting the pre-determined percentage of weights selected from the at least one block to zero. The weight matrix includes a set of weights associated with the layer, the plurality of layers includes a first layer receiving a first input associated with one or more audio feature sequences, and the plurality of layers are executed on one or more processors. The method efficiently accelerates model inference using irregular pruning. ## Claims ### Claim 1 A method for automatic speech recognition, comprising: receiving, by a first layer in a neural network that is executed on one or more processors, a first input associated with one or more audio feature sequences; generating a weight matrix for a layer of a plurality of layers in the neural network, wherein the weight matrix comprises a set of weights associated with the layer, the plurality of layers comprises the first layer receiving the first input, and the plurality of layers are executed on the one or more processors; dividing the weight matrix into a plurality of blocks, each block comprising a plurality of weights, wherein each block has an identical dimension and the identical dimension of each block that is obtained by dividing the weight matrix is determined according to a size of a tensor core unit of the one or more processors; determining a pre-determined percentage for each block, wherein the pre-determined percentage is X %, X is a positive integer; selecting, based on the pre-determined percentage, a sub-set of weights from the plurality of weights in each block for block-wise pruning, wherein a first ratio of a number of the sub-set of weights in each block to a number of the plurality of weights in each block is X %, wherein the sub-set of weights comprise at least one weight; setting the sub-set of weights in each block to zero to generate a block-wise pruned weight matrix, wherein other weights in each block remain non-zero, wherein a second ratio of a number of the other weights in each block to the number of the plurality of weights in each block is 1-X %; and generating, by the one or more processors, a final output for the one or more audio feature sequences based on the block-wise pruned weight matrix. ### Claim 2 (depends on 1) The method of claim 1, wherein the pre-determined percentage is at least 50%. ### Claim 3 (depends on 1) The method of claim 1, wherein the pre-determined percentage of weights are selected by using a heuristic regularization algorithm. ### Claim 4 (depends on 1) The method of claim 1, further comprising: determining, based on a heuristic regularization algorithm, the pre-determined percentage for each block. ### Claim 5 (depends on 4) The method of claim 4, wherein each block has a dimension of 1×4 or 2×2. ### Claim 6 (depends on 1) The method of claim 1, wherein the plurality of layers comprise one or more convolution layers. ### Claim 7 (depends on 6) The method of claim 6, further comprising: transforming a three-dimensional weight tensor of the convolutional layer to the weight matrix. ### Claim 8 An apparatus for automatic speech recognition, comprising: one or more processors; and a memory configured to store instructions executable by the one or more processors; wherein the one or more processors, upon execution of the instructions, are configured to: receive, by a first layer in a neural network that is executed on the one or more processors, a first input associated with one or more audio feature sequences; generate a weight matrix for a layer of a plurality of layers in the neural network, wherein the weight matrix comprises a set of weights associated with the layer, the plurality of layers comprises the first layer receiving a first, and the plurality of layers are executed on the one or more processors; divide the weight matrix into a plurality of blocks, each block comprising a plurality of weights, wherein each block has an identical dimension and the identical dimension of each block that is obtained by dividing the weight matrix is determined according to a size of a tensor core unit of the one or more processors; determine a pre-determined percentage for each block, wherein the pre-determined percentage is X %, X is a positive integer; select, based on the pre-determined percentage, a sub-set of weights from the plurality of weights in each block for block-wise pruning, wherein a first ratio of a number of the sub-set of weights in each block to a number of the plurality of weights in each block is X %, wherein the sub-set of weights comprise at least one weight; set the sub-set of weights in each block to zero to generate a block-wise pruned weight matrix, wherein other weights in each block remain non-zero, wherein a second ratio of a number of the other weights in each block to the number of the plurality of weights in each block is 1-X %; and generating a final output for the one or more audio feature sequences based on the block-wise pruned weight matrix. ### Claim 9 (depends on 8) The apparatus of claim 8, wherein the pre-determined percentage is at least 50%. ### Claim 10 (depends on 8) The apparatus of claim 8, wherein the pre-determined percentage of weights are selected by using a heuristic regularization algorithm. ### Claim 11 (depends on 8) The apparatus of claim 8, wherein the one or more processors are further configured to: determine, based on a heuristic regularization algorithm, the pre-determined percentage for each block. ### Claim 12 (depends on 11) The apparatus of claim 11, wherein each block has a dimension of 1×4 or 2×2. ### Claim 13 (depends on 8) The apparatus of claim 8, wherein the plurality of layers comprise one or more convolution layers. ### Claim 14 (depends on 13) The apparatus of claim 13, wherein the one or more processors are further configured to: transform a three-dimensional weight tensor of the convolutional layer to the weight matrix. ### Claim 15 A non-transitory computer readable storage medium, comprising instructions stored therein, wherein, upon execution of the instructions by one or more processors, the instructions cause the one or more processors to perform acts comprising: receiving, by a first layer in a neural network that is executed on the one or more processors, a first input associated with one or more audio feature sequences; generating a weight matrix for a layer of a plurality of layers in the neural network, wherein the weight matrix comprises a set of weights associated with the layer, the plurality of layers comprises the first layer receiving the first input, and the plurality of layers are executed on the one or more processors; dividing the weight matrix into a plurality of blocks, each block comprising a plurality of weights, wherein each block has an identical dimension and the identical dimension of each block that is obtained by dividing the weight matrix is determined according to a size of a tensor core unit of the one or more processors; determining a pre-determined percentage for each block, wherein the pre-determined percentage is X %, X is a positive integer; selecting, based on the pre-determined percentage, a sub-set of weights from the plurality of weights in each block for block-wise pruning, wherein a first ratio of a number of the sub-set of weights in each block to a number of the plurality of weights in each block is X %, wherein the sub-set of weights comprise at least one weight; setting the sub-set of weights in each block to zero to generate a block-wise pruned weight matrix, wherein other weights in each block remain non-zero, wherein a second ratio of a number of the other weights in each block to the number of the plurality of weights in each block is 1-X %; and generating a final output for the one or more audio feature sequences based on the block-wise pruned weight matrix. ### Claim 16 (depends on 15) The non-transitory computer readable storage medium of claim 15, wherein the pre-determined percentage is at least 50%. ### Claim 17 (depends on 15) The non-transitory computer readable storage medium of claim 15, wherein the instructions cause the one or more processors to perform acts further comprising: determining, based on a heuristic regularization algorithm, the pre-determined percentage for each block. ### Claim 18 (depends on 17) The non-transitory computer readable storage medium of claim 17, wherein each block has a dimension of 1×4 or 2×2. ### Claim 19 (depends on 15) The non-transitory computer readable storage medium of claim 15, wherein the plurality of layers comprise one or more convolution layers. ### Claim 20 (depends on 15) The non-transitory computer readable storage medium of claim 15, wherein the instructions cause the one or more processors to perform acts further comprising: transforming a three-dimensional weight tensor of the convolutional layer to the weight matrix.