# US 10,504,022 B2 · Neural network accelerator with parameters resident on chip > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,504,022 B2 - Document ID: US10504022B2 - Status: Granted - Kind: B2 - Application number: 16/059,686 - Filed: 2018-08-09 - Published: 2019-02-14 - Granted: 2019-12-10 - Earliest claimed: 2017-08-11 - Assignees: Google Inc. - Applicants: Google LLC - Inventors: Olivier Temam; Harshit Khaitan; Ravi Narayanaswami; Dong Hyuk Woo - Web page: https://patentgrep.com/patent/US10504022B2 ## CPC - G06N 3/063: using electronic means - G06F 9/3887: {controlled by a single instruction for multiple data lanes [SIMD]} - G06F 9/3895: {for complex operations, e.g. multidimensional or interleaved address generators, macros} - G06F 13/00: Interconnection of, or transfer of information or other signals between, memories, input/output devices or central processing units (interface circuits for specific input/output devices G06F3/00 {; multiprogram control therefor G06F9/46}; multiprocessor systems G06F15/16 ) - G06F 17/16: Matrix or vector computation {, e.g. matrix-matrix or matrix-vector multiplication, matrix factorization (matrix transposition G06F7/78)} - G06N 3/045: Combinations of networks - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/048: Activation functions - G06N 3/0499: Feedforward networks - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract One embodiment of an accelerator includes a computing unit; a first memory bank for storing input activations and a second memory bank for storing parameters used in performing computations, the second memory bank configured to store a sufficient amount of the neural network parameters on the computing unit to allow for latency below a specified level with throughput above a specified level. The computing unit includes at least one cell comprising at least one multiply accumulate (“MAC”) operator that receives parameters from the second memory bank and performs computations. The computing unit further includes a first traversal unit that provides a control signal to the first memory bank to cause an input activation to be provided to a data bus accessible by the MAC operator. The computing unit performs computations associated with at least one element of a data array, the one or more computations performed by the MAC operator. ## Claims ### Claim 1 An accelerator for accelerating tensor computations, comprising: a computing unit comprising: a first memory bank comprising a first register, the first memory bank configured for storing at least one of input activations or output activations; a second memory bank comprising a second register, the second memory bank configured for storing neural network parameters used in performing computations, the second memory bank configured to store a sufficient amount of the neural network parameters on the computing unit to allow for latency below a specified level with throughput above a specified level for a given neural network (“NN”) model and architecture; at least one cell comprising at least one multiply accumulate (“MAC”) operator that receives parameters from the second memory bank and performs computations; a first tensor traversal unit in data communication with at least the first memory bank, the first tensor traversal unit configured to provide a control signal to the first memory bank to cause an input activation to be provided to a data bus accessible by the MAC operator, and wherein the accelerator performs one or more computations associated with at least one element of a data array, the one or more computations being performed by the MAC operator and comprising, in part, a multiply operation of the input activation received from the data bus and a parameter received from the second memory bank, and wherein the first memory bank, the second memory bank, the first tensor traversal unit, and the at least one MAC operator are located on a same die. ### Claim 2 (depends on 1) The accelerator of claim 1, wherein the second memory bank is configured to store more than 100,000 parameters. ### Claim 3 (depends on 1) The accelerator of claim 1, wherein the second memory bank is configured to store more than 1,000,000 parameters. ### Claim 4 (depends on 1) The accelerator of claim 1, wherein the second memory bank is configured to store more than 100,000,000 parameters. ### Claim 5 (depends on 1) The accelerator of claim 1, wherein the second memory bank comprises SRAM. ### Claim 6 (depends on 1) The accelerator of claim 1, wherein the accelerator comprises a plurality of computing units, each computing unit comprising: a first memory bank for storing at least one of input activations or output activations; a second memory bank for storing neural network parameters used in performing computations, the second memory bank configured to store a sufficient amount of the neural network parameters on the computing unit to allow for latency below a specified level with throughput above a specified level for a given neural network (NN) model and architecture; at least one cell comprising at least one multiply accumulate (“MAC”) operator that receives parameters from the second memory bank and performs computations; a first tensor traversal unit in data communication with at least the first memory bank, the first tensor traversal unit configured to provide a control signal to the first memory bank to cause an input activation to be provided to a data bus accessible by the MAC operator; and wherein the computing unit performs one or more computations associated with at least one element of a data array, the one or more computations being performed by the MAC operator and comprising, in part, a multiply operation of the input activation received from the data bus and a parameter received from the second memory bank. ### Claim 7 (depends on 1) The accelerator of claim 1, wherein the second memory bank comprises 3D SRAM. ### Claim 8 A computer-implemented method for accelerating tensor computations, comprising: sending, by a first memory bank comprising a first register, a first input activation in response to the first memory bank receiving a control signal from a first tensor traversal unit, the first memory bank being disposed in a computing unit, and wherein the first input activation is provided by a data bus that is accessible by at least one cell of the computing unit; receiving, by the at least one cell, one or more parameters from a second memory bank comprising a second register, the second memory bank configured for storing neural network parameters used in performing computations, the second memory bank storing a sufficient amount of the neural network parameters on the computing unit to allow for latency below a specified threshold with throughput above a specified threshold for a given neural network (“NN”) model and architecture, and wherein the at least one cell comprises at least one multiply accumulate (“MAC”) operator; and performing, by the MAC operator, one or more computations associated with at least one element of a data array, wherein the one or more computations comprise, in part, a multiply operation of at least the first input activation accessed from the data bus and at least one parameter received from the second memory bank, wherein the first memory bank, the second memory bank, the first tensor traversal unit, and the at least one MAC operator are located on a same die. ### Claim 9 (depends on 8) The method of claim 8 wherein the second memory bank is configured to store more than 100,000 parameters. ### Claim 10 (depends on 8) The method of claim 8 wherein the second memory bank is configured to store more than 1,000,000 parameters. ### Claim 11 (depends on 8) The method of claim 8 wherein the second memory bank comprises SRAM. ### Claim 12 (depends on 8) The method of claim 8 wherein the second memory bank comprises 3D SRAM. ### Claim 13 (depends on 8) The method of claim 8 further comprising loading the neural network parameters used in performing computations into the second memory. ### Claim 14 A computer-implemented method for accelerating tensor computations, comprising: sending, by a first memory bank comprising a first register, a first input activation in response to the first memory bank receiving a control signal, wherein the first input activation is transmitted by a data bus; receiving, by at least one multiply accumulate (“MAC”) operator, one or more parameters from a second memory bank comprising a second register, the second memory bank configured for storing neural network parameters used in performing computations, the second memory bank storing a sufficient amount of the neural network parameters on the computing unit to allow for latency below a specified threshold and throughput above a specified threshold for a given neural network (“NN”) model and architecture; and performing, by the MAC operator, one or more computations associated with at least one element of a data array, wherein the one or more computations comprise, in part, a multiply operation of at least the first input activation accessed from the data bus and at least one parameter received from the second memory bank, wherein the first memory bank, the second memory bank, and the at least one MAC operator are located on a same die. ### Claim 15 (depends on 14) The method of claim 14, wherein the second memory bank is configured to store more than 100,000 parameters. ### Claim 16 (depends on 14) The method of claim 14, wherein the second memory bank is configured to store more than 1,000,000 parameters. ### Claim 17 (depends on 14) The method of claim 14, wherein the second memory bank is configured to store more than 100,000,000 parameters. ### Claim 18 (depends on 14) The method of claim 14, wherein the second memory bank comprises SRAM. ### Claim 19 (depends on 14) The method of claim 14, wherein the second memory bank comprises 3D SRAM. ### Claim 20 (depends on 14) The method of claim 14, further comprising, loading the neural network parameters used in performing computations into the second memory.