# US 12,001,383 B2 · Deterministic memory for tensor streaming processors > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,001,383 B2 - Document ID: US12001383B2 - Status: Granted - Kind: B2 - Application number: 17/858,493 - Filed: 2022-07-06 - Published: 2023-01-26 - Granted: 2024-06-04 - Earliest claimed: 2021-07-07 - Assignees: Groq, Inc. - Applicants: Groq, Inc. - Inventors: Dennis Charles Abts - Web page: https://patentgrep.com/patent/US12001383B2 ## CPC - G06F 15/80: comprising an array of processing units with common control, e.g. single instruction multiple data processors (G06F15/82 takes precedence {; for correlation function computation G06F17/15}) - G06F 12/023: {Free address space management} - G06F 12/0207: {with multidimensional access, e.g. row/column, matrix} - G06F 12/0284: {Multiple user address space allocation, e.g. using different base addresses (interprocessor communication G06F15/163)} - G06F 12/0813: with a network or matrix configuration - G06F 2212/251: Local memory within processor subsystem - G06F 2212/454: Vector or matrix data - G06N 20/00: Machine learning - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS ## Abstract Embodiments are directed to a deterministic streaming system with one or more deterministic streaming processors each having an array of processing elements and a first deterministic memory coupled to the processing elements. The deterministic streaming system further includes a second deterministic memory with multiple data banks having a global memory address space, and a controller. The controller initiates retrieval of first data from the data banks of the second deterministic memory as a first plurality of streams, each stream of the first plurality of streams streaming toward a respective group of processing elements of the array of processing elements. The controller further initiates writing of second data to the data banks of the second deterministic memory as a second plurality of streams, each stream of the second plurality of streams streaming from the respective group of processing elements toward a respective data bank of the second deterministic memory. ## Claims ### Claim 1 A deterministic streaming system comprising: one or more deterministic streaming processors, each deterministic streaming processor including: an array of processing elements, and a first deterministic memory communicatively coupled to the array of processing elements; a second deterministic memory communicatively coupled with the array of processing elements, the second deterministic memory comprising a plurality of data banks having a global memory address space for the one or more deterministic streaming processors, wherein the first deterministic memory comprises a static memory and the second deterministic memory comprises a dynamic memory; and a controller communicatively coupled with the second deterministic memory and configured to: initiate retrieval of first data from the plurality of data banks of the second deterministic memory as a first plurality of streams, each stream of the first plurality of streams streaming toward a respective group of processing elements of the array of processing elements, and initiate writing of second data to the plurality of data banks of the second deterministic memory as a second plurality of streams, each stream of the second plurality of streams streaming from the respective group of processing elements toward a respective data bank of the second deterministic memory. ### Claim 2 (depends on 1) The deterministic streaming system of claim 1, wherein the controller is further configured to assign each stream of the first plurality of streams and the second plurality of streams to a respective pair of pseudo channels of a plurality of pseudo channels of the second deterministic memory. ### Claim 3 (depends on 2) The deterministic streaming system of claim 2, wherein the respective pair of pseudo channels form an address channel of the second deterministic memory. ### Claim 4 (depends on 2) The deterministic streaming system of claim 2, wherein each pair of pseudo channels of the plurality of pseudo channels is mapped to the respective group of processing elements. ### Claim 5 (depends on 1) The deterministic streaming system of claim 1, wherein the second deterministic memory is configured for simultaneous access of the plurality of data banks. ### Claim 6 (depends on 1) The deterministic streaming system of claim 1, wherein the first deterministic memory is communicatively coupled to each group of processing elements in the array of processing elements via a respective superlane of the deterministic streaming processor. ### Claim 7 (depends on 1) The deterministic streaming system of claim 1, wherein: each stream of the first plurality of streams is configured to stream toward the respective group of processing elements via a respective superlane of the deterministic streaming processor; and each stream of the second plurality of streams is configured to stream toward the second deterministic memory via the respective superlane. ### Claim 8 (depends on 1) The deterministic streaming system of claim 1, wherein the dynamic memory comprises one or more three-dimensional stacks of a high bandwidth memory devices. ### Claim 9 (depends on 1) The deterministic streaming system of claim 1, wherein the static memory is a static random access memory, and the dynamic memory comprises one or more three-dimensional stacks of a dynamic random access memory devices. ### Claim 10 (depends on 1) The deterministic streaming system of claim 1, wherein: the one or more deterministic streaming processors comprise a plurality of deterministic streaming processors organized as a node of the deterministic streaming system; and the controller is further configured to: assign each of the deterministic streaming processors to a respective pair of pseudo channels of a plurality of pseudo channels of the second deterministic memory, and initiate streaming of data between each of the deterministic streaming processors and a respective data bank of the plurality of data banks of the second deterministic memory associated with the respective pair of pseudo channels. ### Claim 11 (depends on 1) The deterministic streaming system of claim 1, wherein: the deterministic streaming system further comprises a plurality of nodes organized as a rack of the deterministic streaming system, each of the nodes comprising a plurality of deterministic streaming processors; and the controller is further configured to: assign each of the nodes to a respective pair of pseudo channels of a plurality of pseudo channels of the second deterministic memory, and initiate streaming of data between each of the nodes and a respective data bank of the plurality of data banks of the second deterministic memory associated with the respective pair of pseudo channels. ### Claim 12 (depends on 1) The deterministic streaming system of claim 1, wherein: the deterministic streaming system further comprises a plurality of racks, each of the racks comprises a plurality of nodes, each of the nodes comprising a plurality of deterministic streaming processors; and the controller is further configured to: assign each of the racks to a respective pair of pseudo channels of a plurality of pseudo channels of the second deterministic memory, and initiate streaming of data between each of the racks and a respective data bank of the plurality of data banks of the second deterministic memory associated with the respective pair of pseudo channels. ### Claim 13 (depends on 1) The deterministic streaming system of claim 1, further comprising a compiler configured to: manage initiation of the retrieval of the first data at a predetermined first time instant; and manage initiation of the writing of the second data at a predetermined second time instant. ### Claim 14 A method of deterministic computing at a deterministic streaming system, the method comprising: initiating streaming of data between a first deterministic memory of a deterministic streaming processor of the deterministic streaming system and an array of processing elements of the deterministic streaming processor; initiating retrieval of first data from a plurality of data banks of a second deterministic memory of the deterministic streaming system as a first plurality of streams, each stream of the first plurality of streams streaming toward a respective group of processing elements of the array of processing elements, the second deterministic memory having a global memory address space for one or more deterministic streaming processors of the deterministic streaming system; and initiating writing of second data to the plurality of data banks as a second plurality of streams, each stream of the second plurality of streams streaming from the respective group of processing elements toward a respective data bank of the second deterministic memory, wherein the one or more deterministic streaming processors comprise a plurality of deterministic streaming processors organized as a node of the deterministic streaming system, and the method further comprising: assigning each of the deterministic streaming processors to a respective pair of pseudo channels of a plurality of pseudo channels of the second deterministic memory; and initiating streaming of data between each of the deterministic streaming processors and a respective data bank of the plurality of data banks of the second deterministic memory associated with the respective pair of pseudo channels. ### Claim 15 (depends on 14) The method of claim 14, further comprising: assigning each stream of the first plurality of streams and the second plurality of streams to a respective pair of pseudo channels of the plurality of pseudo channels of the second deterministic memory; and mapping each pair of pseudo channels of the plurality of pseudo channels to the respective group of processing elements. ### Claim 16 (depends on 14) The method of claim 14, further comprising: initiating streaming of each stream of the first plurality of streams toward the respective group of processing elements via a respective superlane of the deterministic streaming processor; and initiating streaming of each stream of the second plurality of streams toward the second deterministic memory via the respective superlane. ### Claim 17 (depends on 14) The method of claim 14, wherein a plurality of nodes, including the node, are organized as a rack of the deterministic streaming system. ### Claim 18 A non-transitory computer-readable storage medium having stored thereon executable instructions, which when executed by a computer processor cause the computer processor to: initiate streaming of data between a first deterministic memory of a deterministic streaming processor of a deterministic streaming system and an array of processing elements of the deterministic streaming processor; initiate retrieval of first data from a plurality of data banks of a second deterministic memory of the deterministic streaming system as a first plurality of streams, each stream of the first plurality of streams streaming toward a respective group of processing elements of the array of processing elements, the second deterministic memory having a global memory address space for one or more deterministic streaming processors of the deterministic streaming system; and initiate writing of second data to the plurality of data banks as a second plurality of streams, each stream of the second plurality of streams streaming from the respective group of processing elements toward a respective data bank of the second deterministic memory, wherein the one or more deterministic streaming processors comprise a plurality of deterministic streaming processors organized as a node of the deterministic streaming system, and the method further comprising: assigning each of the deterministic streaming processors to a respective pair of pseudo channels of a plurality of pseudo channels of the second deterministic memory; and initiating streaming of data between each of the deterministic streaming processors and a respective data bank of the plurality of data banks of the second deterministic memory associated with the respective pair of pseudo channels. ### Claim 19 (depends on 18) The non-transitory computer-readable storage medium of claim 18, wherein a plurality of nodes, including the node, are organized as a rack of the deterministic streaming system. ### Claim 20 (depends on 18) The non-transitory computer-readable storage medium of claim 18, wherein the instructions, when expected by the computer processor further cause the computer processor to: further comprising: initiate streaming of each stream of the first plurality of streams toward the respective group of processing elements via a respective superlane of the deterministic streaming processor; and initiate streaming of each stream of the second plurality of streams toward the second deterministic memory via the respective superlane.