# US 11,907,827 B2 · Schedule-aware tensor distribution module > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,907,827 B2 - Document ID: US11907827B2 - Status: Granted - Kind: B2 - Application number: 16/456,707 - Filed: 2019-06-28 - Published: 2020-12-31 - Granted: 2024-02-20 - Earliest claimed: 2019-06-28 - Assignees: Intel Corporation - Applicants: Intel Corporation - Inventors: Gautham Chinya; Huichu Liu; Arnab Raha; Debabrata Mohapatra; Cormac Brick; Lance Hacking - Web page: https://patentgrep.com/patent/US11907827B2 ## CPC - G06N 3/063: using electronic means - G06F 9/3814: {Implementation provisions of instruction buffers, e.g. prefetch buffer; banks} - G06F 9/3877: {using a secondary processor, e.g. coprocessor (peripheral processor G06F13/12)} - G06F 9/4498: {Finite state machines} - G06F 9/5027: {the resource being a machine, e.g. CPUs, Servers, Terminals} - G06N 5/04: Inference or reasoning models - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract Methods and systems include a neural network system that includes a neural network accelerator. The neural network accelerator includes multiple processing engines coupled together to perform arithmetic operations in support of an inference performed using the deep neural network system. The neural network accelerator also includes a schedule-aware tensor data distribution circuitry or software that is configured to load tensor data into the multiple processing engines in a load phase, extract output data from the multiple processing engines in an extraction phase, reorganize the extracted output data, and store the reorganized extracted output data to memory. ## Claims ### Claim 1 A neural network accelerator, comprising: a memory; a plurality of processing engines coupled together and configured to perform arithmetic operations in support of an inference performed using the neural network accelerator, wherein the plurality of processing engines are implemented using a plurality of processing elements; and a schedule-aware tensor data distribution circuitry configured to: load tensor data into the plurality of processing engines in a load phase; extract output data from the plurality of processing engines in an extraction phase, wherein extracting the output data from the plurality of processing engines may be performed in a row-wise or column-wise organization; reorganize the extracted output data based at least in part on a schedule for a next layer after a current layer to output the extracted output data by reshaping the extracted output data for storage in the memory to reduce a number of accesses to the memory for the next layer by changing a shape of how the extracted output data is stored in the memory based at least in part on a type of the next layer, wherein changing the shape comprises reorganizing the extracted output data to a column-wise organization from the row-wise organization or to the row-wise organization from the column-wise organization based on a specification of the next layer; and store the reorganized extracted output data to the memory. ### Claim 2 (depends on 1) The neural network accelerator of claim 1, wherein the memory comprises a static random-access memory (SRAM). ### Claim 3 (depends on 1) The neural network accelerator of claim 1, wherein the schedule-aware tensor data distribution circuitry comprises buffers, and the schedule-aware tensor data distribution circuitry is configured to use the buffers in the reorganization of the extracted output data before storing the reorganized extracted output data in the memory. ### Claim 4 (depends on 1) The neural network accelerator of claim 1, wherein the schedule-aware tensor data distribution circuitry comprises a drain finite state machine that performs the reorganization of the extracted output data. ### Claim 5 (depends on 1) The neural network accelerator of claim 1, wherein the schedule-aware tensor data distribution circuitry comprises a load finite state machine that performs loading of the tensor data into the plurality of processing engines. ### Claim 6 (depends on 1) The neural network accelerator of claim 1, wherein the tensor data comprises input activations or feature maps used to control how input data is input to the plurality of processing engines. ### Claim 7 (depends on 1) The neural network accelerator of claim 1, wherein the tensor data comprises weights or filter values that are used to weight computations of the plurality of processing engines. ### Claim 8 (depends on 1) The neural network accelerator of claim 1, wherein the plurality of processing elements comprises respective plurality of multiply accumulate circuitries. ### Claim 9 A deep neural network system, comprising: one or more neural network accelerators comprising: a plurality of processing engines coupled together and configured to perform arithmetic operations in support of an inference performed using the deep neural network system, wherein the plurality of processing engines are implemented using a plurality of processing elements; a schedule-aware tensor data distribution circuitry, comprising: a load finite state machine configured to load tensor data into the plurality of processing engines in a load phase; a drain finite state machine that is configured to: extract output data from the plurality of processing engines in an extraction phase; adjust the extracted output data based at least in part on a schedule for a next layer after a current layer to output the extracted output data by reshaping the extracted output data for storage in memory to reduce a number of accesses to the memory for the next layer by changing a shape of how the extracted output data is stored in the memory based at least in part on a type of the next layer; and store the adjusted extracted output data to a plurality of memory banks. ### Claim 10 (depends on 9) The deep neural network system of claim 9, wherein the drain finite state machine comprises a plurality of memory buffers. ### Claim 11 (depends on 10) The deep neural network system of claim 10, wherein the plurality of processing engines are arranged in an array of rows and columns, and a number of memory buffers of the plurality of memory buffers equals a number of columns of processing engines of the plurality of processing engines. ### Claim 12 (depends on 11) The deep neural network system of claim 11, wherein a number of memory banks of the plurality of memory banks is equal to the number of memory buffers of the plurality of memory buffers, wherein each memory buffer is dedicated to a corresponding memory bank of the plurality of memory banks. ### Claim 13 (depends on 9) The deep neural network system of claim 9, wherein the adjustment of the extracted output data addresses a difference in extraction and loading schemes between layers of the inference to reduce energy consumption by the one or more neural network accelerators due to the reduced number of memory accesses. ### Claim 14 (depends on 13) The deep neural network system of claim 13, wherein reducing energy consumption comprises reducing a number of writes to the plurality of memory banks due to the difference in extraction and loading schemes between the layers of the inference. ### Claim 15 (depends on 14) The deep neural network system of claim 14, wherein the plurality of memory banks comprises a plurality of static random-access memory banks. ### Claim 16 A method, comprising: for a plurality of layers of a deep neural network inference: updating configuration descriptor registers used to describe a schedule for a current layer of the plurality of layers; based at least in part on the configuration descriptor registers, loading tensor data into a plurality of processing engines; using the loaded tensor data, performing computations using the plurality of processing engines for the deep neural network inference; after the computations are performed, extracting output data; and reorganizing the extracted output data to memory banks based on a schedule for a next layer of the plurality of layers by reshaping the extracted output data for storage in the memory banks to reduce a number of accesses to the memory banks for the next layer by changing a shape of how the extracted output data is stored in the memory banks based at least in part on a type of the next layer. ### Claim 17 (depends on 16) The method of claim 16, wherein for the plurality of layers of the deep neural network inference, loading the reorganized output data to the plurality of processing engines in a loading phase for the next layer. ### Claim 18 A neural network accelerator, comprising: a memory; a plurality of processing engines coupled together and configured to perform arithmetic operations in support of an inference performed using the neural network accelerator, wherein the plurality of processing engines are implemented using a plurality of processing elements; and a distribution circuitry configured to: load tensor data into the plurality of processing engines in a load phase; extract output data from the plurality of processing engines in an extraction phase, wherein extracting the output data from the plurality of processing engines may be performed in a row-wise or column-wise organization; reorganize the extracted output data based at least in part on a schedule for a next layer after a current layer to output the extracted output data by reshaping the extracted output data for storage in the memory to reduce a number of accesses to the memory for the next layer by changing a shape of how the extracted output data is stored in the memory based at least in part on a type of the next layer, wherein changing the shape comprises reorganizing the extracted output data to a column-wise organization from the row-wise organization or to the row-wise organization from the column-wise organization based on a specification of a next layer; and store the reorganized extracted output data to the memory.