# US 2021/0241095 A1 · Deep learning processing apparatus and method, device and storage medium > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2021/0241095 A1 - Document ID: US20210241095A1 - Status: Published application - Kind: A1 - Application number: 17/017,600 - Filed: 2020-09-10 - Published: 2021-08-05 - Earliest claimed: 2020-02-05 - Assignees: Beijing Baidu Netcom Science Technology Co., Ltd. - Applicants: Beijing Baidu Netcom Science and Technology Co., Ltd. - Inventors: Xiaozhang Gong; Jian Ouyang; Jing Wang; Wei Qi - Web page: https://patentgrep.com/patent/US20210241095A1 ## CPC - G06F 17/16: Matrix or vector computation {, e.g. matrix-matrix or matrix-vector multiplication, matrix factorization (matrix transposition G06F7/78)} - G06N 3/063: using electronic means - G06F 7/5443: {Sum of products (for applications thereof, see the relevant places, e.g. G06F17/10, H03H17/00)} - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/084: Backpropagation, e.g. using gradient descent - G06N 3/09: Supervised learning - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS ## Abstract Embodiments of the present disclosure propose a deep learning processing apparatus and method, device and storage medium, relating to the field of artificial intelligence. A deep learning processing apparatus includes: at least one matrix multiply-add module, configured to perform a matrix multiply-add operation of a convolution kernel parameter value matrix of a convolutional layer in a convolutional neural network and a first error gradient value matrix to obtain a plurality of intermediate matrices; a storage apparatus, configured to store the plurality of intermediate matrices without reshaping elements in the plurality of intermediate matrices; and a plurality of matrix accumulation modules, configured to read the plurality of intermediate matrices from the storage apparatus and perform a matrix accumulation operation based on the plurality of intermediate matrices according to a convolution scheme of the convolutional layer in parallel, to obtain a second error gradient value matrix for the convolutional layer. ## Claims ### Claim 1 A deep learning processing apparatus, comprising: at least one matrix multiply-add module, configured to perform a matrix multiply-add operation of a convolution kernel parameter value matrix of a convolutional layer in a convolutional neural network and a first error gradient value matrix to obtain a plurality of intermediate matrices; a storage apparatus, configured to store the plurality of intermediate matrices without reshaping elements in the plurality of intermediate matrices; and a plurality of matrix accumulation modules, configured to read the plurality of intermediate matrices from the storage apparatus and perform a matrix accumulation operation based on the plurality of intermediate matrices according to a convolution scheme of the convolutional layer in parallel, to obtain a second error gradient value matrix for the convolutional layer. ### Claim 2 (depends on 1) The deep learning processing apparatus according to claim 1, wherein the plurality of intermediate matrices are associated with a plurality of channels of an input of the convolutional layer, and the second error gradient value matrix comprises a plurality of channel matrices for the plurality of channels, each of the plurality of matrix accumulation modules is configured to: accumulate an intermediate matrix, in the plurality of intermediate matrices, associated with a channel of the plurality of channels, on a parallel path of a plurality of parallel paths corresponding to the plurality of channels to obtain the channel matrix for the channel; or accumulate, at each accumulation, one of the plurality of intermediate matrices to an intermediate result for one of the plurality of channel matrices. ### Claim 3 (depends on 1) The deep learning processing apparatus according to claim 1, wherein the storage apparatus is configured to store the second error gradient value matrix in a target storage location, the second error gradient value matrix having an all zero value before the matrix accumulation operation; and wherein the plurality of matrix accumulation modules are configured to accumulate the plurality of intermediate matrices in parallel with the second error gradient value matrix having an initial value stored in the target storage location. ### Claim 4 (depends on 1) The deep learning processing apparatus according to claim 1, wherein at least one matrix accumulation module of the plurality of matrix accumulation modules comprises: an addition operation unit, configured to receive a first element and a second element in to-be-accumulated matrices and accumulate the first element and the second element to obtain an accumulated element; a data writing unit, configured to write the accumulated element to the storage apparatus; a first cache area, configured to cache the accumulated element, a writing speed of the first cache area being faster than a writing speed of the storage apparatus, the accumulated element being cached in the first cache area for a plurality of clock cycles; and a bypass module, configured to, in response to determining that the accumulated element cannot be read from the storage apparatus when the addition operation unit is to perform accumulation of a third element and the accumulated element, cause the accumulated element cached in the first cache area to be provided to the addition operation unit. ### Claim 5 (depends on 1) The deep learning processing apparatus according to claim 1, wherein the storage apparatus comprises an on-chip random access memory. ### Claim 6 (depends on 1) The deep learning processing apparatus according to claim 1, further comprising: a second cache area, configured to cache a first intermediate operation result generated by the at least one matrix multiply-add module during the matrix multiply-add operation and cache second intermediate operation results generated by the plurality of matrix accumulation modules during the matrix accumulation operation. ### Claim 7 (depends on 6) The deep learning processing apparatus according to claim 6, further comprising: a vector calculation module, configured to perform at least one of: zeroing a storage area for storing the first intermediate operation result and the second intermediate operation results in the second cache area at a vector level; and writing a final operation result of the matrix accumulation operation selectively to the storage apparatus of the deep learning processing apparatus as at least a part of the second error gradient value matrix at the vector level. ### Claim 8 (depends on 1) The deep learning processing apparatus according to claim 1, further comprising: an instruction processing module, configured to acquire an instruction set for a convolution direction operation of the convolutional layer in the CNN from a general-purpose processing apparatus and parse the instruction set; and a control signal generation module, configured to generate a control signal for the at least one matrix multiply-add module and the plurality of matrix accumulation modules based on the parsed instruction set. ### Claim 9 (depends on 1) The deep learning processing apparatus according to claim 1, wherein the at least one matrix multiply-add module comprises a plurality of matrix multiply-add modules, and configured to perform the matrix multiply-add operation in parallel. ### Claim 10 A method for performing deep learning processing, comprising: causing at least one matrix multiply-add module of a deep learning processing apparatus to perform a matrix multiply-add operation of a convolution kernel parameter value matrix of a convolutional layer in a convolutional neural network and a first error gradient value matrix to obtain a plurality of intermediate matrices; storing the plurality of intermediate matrices to a storage apparatus without reshaping elements in the plurality of intermediate matrices; reading the plurality of intermediate matrices from the storage apparatus; and causing a plurality of matrix accumulation modules of the deep learning processing apparatus to perform a matrix accumulation operation based on the plurality of intermediate matrices according to a convolution scheme of the convolutional layer in parallel, to obtain a second error gradient value matrix for the convolutional layer. ### Claim 11 (depends on 10) The method according to claim 10, wherein the plurality of intermediate matrices are associated with a plurality of channels of an input of the convolutional layer, and the second error gradient value matrix comprises a plurality of channel matrices for the plurality of channels, wherein causing the plurality of matrix accumulation modules to perform the matrix accumulation operation based on the plurality of intermediate matrices according to the convolution scheme of the convolutional layer in parallel comprises at least one of: causing each matrix accumulation module in the plurality of matrix accumulation modules to accumulate an intermediate matrix, in the plurality of intermediate matrices, associated with a channel of the plurality of channels, on a parallel path of a plurality of parallel paths corresponding to the plurality of channels to obtain the channel matrix for the channel; or causing each matrix accumulation module in the plurality of matrix accumulation modules to accumulate, at each accumulation, one of the plurality of intermediate matrices to an intermediate result for one of the plurality of channel matrices. ### Claim 12 (depends on 10) The method according to claim 10, wherein causing the plurality of matrix accumulation modules to perform the matrix accumulation operation based on the plurality of intermediate matrices according to the convolution scheme of the convolutional layer in parallel comprises causing at least one matrix accumulation module of the plurality of matrix accumulation modules to: accumulating a first element and a second element to obtain an accumulated element; writing the accumulated element to the storage apparatus; caching the accumulated element to a first cache area, a writing speed of the first cache area being faster than a writing speed of the storage apparatus, the accumulated element being cached in the first cache area for a plurality of clock cycles; and causing the accumulated element cached in the first cache area to be provided for accumulation with the third element, in response to determining that the accumulated element cannot be read from the storage apparatus when an addition operation unit is to perform accumulation of a third element and the accumulated element. ### Claim 13 (depends on 10) The method according to claim 10, further comprising: caching a first intermediate operation result generated by the at least one matrix multiply-add module during the matrix multiply-add operation and second intermediate operation results generated by the plurality of matrix accumulation modules during the matrix accumulation operation to a second cache area. ### Claim 14 (depends on 13) The method according to claim 13, further comprising causing a vector calculation module of the deep learning processing apparatus to perform at least one of: zeroing a storage area for storing the first intermediate operation result and the second intermediate operation results in the second cache area at a vector level; and writing a final operation result of the matrix accumulation operation selectively to the storage apparatus of the deep learning processing apparatus as at least a part of the second error gradient value matrix at the vector level. ### Claim 15 (depends on 10) The method according to claim 10, further comprising: acquiring an instruction set for a convolution direction operation of the convolutional layer in the CNN from a general-purpose processing apparatus and parsing the instruction set; and generating a control signal for the at least one matrix multiply-add module and the plurality of matrix accumulation modules based on the parsed instruction set. ### Claim 16 (depends on 10) The method according to claim 10, wherein the at least one matrix multiply-add module comprises a plurality of matrix multiply-add modules, and causing the at least one matrix multiply-add module to perform the matrix multiply-add operation comprises: causing the plurality of matrix multiply-add modules to perform the matrix multiply-add operation in parallel. ### Claim 17 (depends on 1) An electronic device, comprising: at least one deep learning processing apparatus according to claim 1; and at least one general-purpose processing apparatus, coupled to the at least one deep learning processing apparatus and configured to provide the at least one deep learning processing apparatus with an instruction for performing training of a convolutional neural network. ### Claim 18 (depends on 10) A computer readable storage medium, storing a computer program thereon, the program, when executed by a processor, implements the method according to claim 10.