# US 2023/0273879 A1 · Critical Stage Optimization for Reconfigurable Architectures

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2023/0273879 A1
- Document ID: US20230273879A1
- Status: Published application
- Kind: A1
- Application number: 18/115,118
- Filed: 2023-02-28
- Published: 2023-08-31
- Earliest claimed: 2022-02-28
- Assignees: SambaNova Systems, Inc.
- Applicants: SambaNova Systems, Inc.
- Inventors: Adam BORDELON; David Alan KOEPLINGER
- Web page: https://patentgrep.com/patent/US20230273879A1

## CPC

- G06F 8/4441: {Reducing the execution time required by the program code}
- G06N 3/063: using electronic means
- G06F 15/7867: {with reconfigurable architecture}
- G06F 15/825: {Dataflow computers}
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

A method for reducing latency and increasing throughput in a reconfigurable computing system includes receiving a user program for execution on a reconfigurable dataflow computing system, comprising a grid of compute units and grid of memory units interconnected with a switching array. The user program includes multiple tensor-based algebraic expressions that are converted to an intermediate representation comprising multiple stages. Each stage includes one or more logical operations executable via dataflow through compute units, and each stage is preceded by and followed by a buffer, each buffer corresponding to one or more memory units. The method includes detecting a memory mapping operation within a critical stage and moving the memory mapping operation to an adjacent stage, wherein the memory mapping operation is executable by memory units within the adjacent stage and dataflow through the buffer is controlled by one or more memory units within the grid of memory units.

## Claims

### Claim 1

A system for reducing latency and increasing throughput in reconfigurable dataflow processors, the system comprising:
a host computer comprising an optimization module configured to conduct a method comprising:
receiving a user program for execution on a reconfigurable dataflow computing system, the reconfigurable dataflow computing system comprising a grid of compute units and a grid of memory units interconnected with a switching array, the user program comprising a plurality of tensor-based algebraic expressions;
converting the plurality of tensor-based algebraic expressions to an intermediate representation comprising a plurality of stages, each stage comprising one or more logical operations executable via dataflow through one or more compute units of the grid of compute units, each stage preceded by and followed by a buffer, each buffer corresponding to one or more memory units within the grid of memory units;
detecting a memory mapping operation within a critical stage;
moving the memory mapping operation to an adjacent stage; and
wherein the memory mapping operation is executable by the one or more memory units within the adjacent stage and wherein dataflow through the buffer is controlled by one or more memory units within the grid of memory units.

### Claim 2 (depends on 1)

The system of claim 1, wherein the memory mapping operation comprises one or more of a transpose operation, a reshape operation, a layout transformation, a roll operation, a permutation operation, a slice operation and a tile operation.

### Claim 3 (depends on 1)

The system of claim 1, wherein the one or more logical operations correspond to one or more template library functions.

### Claim 4 (depends on 1)

The system of claim 1, wherein the critical stage comprises a logical operation selected from one or more of matrix multiplication operation, a batch normalization operation, a batch Cholesky operation, and a layer normalization operation.

### Claim 5 (depends on 1)

The system of claim 1, wherein the adjacent stage comprises a logical operation selected from a ReLU operation, a Sigmoid operation, and a Hyperbolic Tangent operation.

### Claim 6 (depends on 1)

The system of claim 1, wherein the logical operations are represented as dataflow statements or compute graph nodes.

### Claim 7 (depends on 1)

The system of claim 1, wherein moving the memory mapping operation to an adjacent stage exposes further optimizations.

### Claim 8 (depends on 7)

The system of claim 7, wherein the further optimizations include Buffer-Buffer Fusion, Buffer/View/Transform Fusion and Prepone/Postpone Views.

### Claim 9

A method for reducing latency and increasing throughput in a reconfigurable computing system, the method comprising:
receiving a user program for execution on a reconfigurable dataflow computing system, the reconfigurable dataflow computing system comprising a grid of compute units and a grid of memory units interconnected with a switching array, the user program comprising a plurality of tensor-based algebraic expressions;
converting the plurality of tensor-based algebraic expressions to an intermediate representation comprising a plurality of stages, each stage comprising one or more logical operations executable via dataflow through one or more compute units of the grid of compute units, each stage preceded by and followed by a buffer, each buffer corresponding to one or more memory units within the grid of memory units;
detecting a memory mapping operation within a critical stage;
moving the memory mapping operation to an adjacent stage; and
wherein the memory mapping operation is executable by the one or more memory units within the adjacent stage and wherein dataflow through the buffer is controlled by one or more memory units within the grid of memory units.

### Claim 10 (depends on 9)

The method of claim 9, wherein the memory mapping operation comprises one or more of a transpose operation, a reshape operation, a layout transformation, a roll operation, a permutation operation, a slice operation and a tile operation.

### Claim 11 (depends on 9)

The method of claim 9, wherein the one or more logical operations correspond to one or more template library functions.

### Claim 12 (depends on 9)

The method of claim 9, wherein the critical stage comprises a logical operation selected from one or more of matrix multiplication operation, a batch normalization operation, a batch Cholesky operation, and a layer normalization operation.

### Claim 13 (depends on 9)

The method of claim 9, wherein the adjacent stage comprises a logical operation selected from a ReLU operation, a Sigmoid operation, and a Hyperbolic Tangent operation.

### Claim 14 (depends on 9)

The method of claim 9, wherein the logical operations are represented as dataflow statements or compute graph nodes.

### Claim 15 (depends on 9)

The method of claim 9, wherein moving the memory mapping operation to an adjacent stage exposes further optimizations.

### Claim 16 (depends on 15)

The method of claim 15, wherein the further optimizations include Buffer-Buffer Fusion, Buffer/View/Transform Fusion and Prepone/Postpone Views.

### Claim 17

A computer program product comprising a computer readable storage medium having program instructions embodied therewith, wherein the computer readable storage medium is not a transitory signal per se, wherein the program instructions are executable by a processor to cause the processor to conduct a method comprising:
receiving a user program for execution on a reconfigurable dataflow computing system, the reconfigurable dataflow computing system comprising a grid of compute units and a grid of memory units interconnected with a switching array, the user program comprising a plurality of tensor-based algebraic expressions;
converting the plurality of tensor-based algebraic expressions to an intermediate representation comprising a plurality of stages, each stage comprising one or more logical operations executable via dataflow through one or more compute units of the grid of compute units, each stage preceded by and followed by a buffer, each buffer corresponding to one or more memory units within the grid of memory units;
detecting a memory mapping operation within a critical stage;
moving the memory mapping operation to an adjacent stage; and
wherein the memory mapping operation is executable by the one or more memory units within the adjacent stage and wherein dataflow through the buffer is controlled by one or more memory units within the grid of memory units.
