# US 2023/0066518 A1 · Heterogeneous architecture for depthwise-seperable convolution based neural network computation acceleration

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2023/0066518 A1
- Document ID: US20230066518A1
- Status: Published application
- Kind: A1
- Application number: 17/460,584
- Filed: 2021-08-30
- Published: 2023-03-02
- Earliest claimed: 2021-08-30
- Assignees: Black Sesame International Holding Limited
- Applicants: Black Sesame International Holding Limited
- Inventors: Yi Wang; Zheng Qi
- Web page: https://patentgrep.com/patent/US20230066518A1

## CPC

- G06N 3/063: using electronic means
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/0495: Quantised networks; Sparse networks; Compressed networks
- G06N 3/048: Activation functions
- G06N 3/082: modifying the architecture, e.g. adding, deleting or silencing nodes or connections
- Y02D 10/00: Energy efficient computing, e.g. low power processors, power management or thermal management
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- Y02D: CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE

## Abstract

The present invention relates to a method and a system for performing depthwise separable convolution on an input data in a convolutional neural network. The invention utilizes a heterogeneous architecture with a number of MAC arrays including 1D MAC arrays and 2D MAC arrays with a Winograd conversion logic to perform depthwise separable convolution. The depthwise separable convolution uses less weight parameters and thus less multiplications while it obtains the same computation results as the traditional convolution.

## Claims

### Claim 1

A method for achieving high utilization of a neural network based computation using depthwise-separable convolution, wherein the method comprising:
performing a point-wise convolution with a two dimensional MAC array on an input data for generating a first output within a spatial domain;
performing a depth-wise convolution with a one dimensional MAC array on the first output for generating a second output within a Winograd domain;
performing a point-wise convolution with the two dimensional MAC array on the second output for generating a final output within the spatial domain from the Winograd domain;
splitting the final output into a plurality of units by a processing unit;
stripping one or more units of the plurality of units by the processing unit; and
accumulating the one or more units of the plurality of units for computing the depthwise-separable convolution.

### Claim 2 (depends on 1)

The method in accordance with claim 1, wherein the first output is distributed and stored in a pipeline buffer.

### Claim 3 (depends on 1)

The method in accordance with claim 1, wherein processing of the first output to the second output from the point-wise convolution to the depth-wise convolution is performed by using a number of buffers.

### Claim 4 (depends on 3)

The method in accordance with claim 3, wherein the number of buffers form a pseudo pipeline.

### Claim 5 (depends on 1)

The method in accordance with claim 1, wherein the conversion from the spatial domain to the Winograd domain is performed by an adder tree structure.

### Claim 6 (depends on 6)

The method in accordance with claim 6, wherein the conversion from the Winograd domain to the spatial domain is performed by an adder tree structure.

### Claim 7 (depends on 5, 6)

The method in accordance to claim 5 and claim 6, wherein the adder tree structure supports different kernel sizes.

### Claim 8 (depends on 1)

The method for achieving high utilization of a neural network based computation using depthwise-separable convolution in accordance with claim 1, wherein the neural network is of a BST chip.

### Claim 9 (depends on 9)

The BST chip in accordance with claim 9 is A500, A2000 and A1000.

### Claim 10 (depends on 1)

The method in accordance with claim 1, wherein stripping the one or more units allow processing depthwise-separable convolution by a single DDR load and a single DDR store.

### Claim 11 (depends on 10)

The single unit in accordance with claim 10, wherein processing depthwise-separable convolution by the single DDR load and the single DDR store limits access to the DDR.

### Claim 12 (depends on 1)

The method in accordance with claim 1, wherein the neural network architecture is a heterogeneous architecture.

### Claim 13 (depends on 1)

The method in accordance with claim 1, wherein the depthwise-separable convolution reduces computation complexity and power demand.

### Claim 14

A heterogeneous architecture for depthwise-separable convolution based neural network computation acceleration, wherein the heterogeneous architecture comprising:
a plurality of MAC arrays to perform depthwise-separable convolution, wherein the depthwise-separable convolution, further wherein the plurality of MAC arrays comprising:
one or more two-dimensional MAC-arrays for performing a point-wise convolution in a spatial domain, wherein the one or more two-dimensional MAC-arrays performs the point-wise convolution on an input data to generates a first output; and
one or more one-dimensional MAC-arrays for performing a depthwise convolution in a Winograd domain, wherein the one or more one-dimensional MAC-arrays performs the Winograd convolution on the first output to generate a second output, further wherein the one or more two-dimensional MAC-arrays performs the point-wise convolution on the second output with an adder tree structure to generate a final output:
a processing unit, wherein the processing unit comprising:
a splitting unit, wherein the splitting unit splits the final output into a plurality of tiles;
a stripping unit, wherein the stripping unit strips one or more units of the plurality of tiles; and
an accumulator, wherein the accumulator accumulates the one or more units of the plurality of tiles for computing the depthwise-separable convolution.

### Claim 15

A computer program product comprising a computer useable medium having computer program logic for enabling at least one processor in a computer system for performing a high utilization of a neural network based computation using depthwise-separable convolution, said computer program logic comprising:
performing a point-wise convolution with a two dimensional MAC array on an input data for generating a first output within a spatial domain;
performing a depth-wise convolution with a one dimensional MAC array on the first output for generating a second output within a Winograd domain;
performing a point-wise convolution with the two dimensional MAC array on the second output for generating a final output within the spatial domain from the Winograd domain;
splitting the final output into a plurality of units by a processing unit;
stripping one or more units of the plurality of units by the processing unit; and
accumulating the one or more units of the plurality of units for computing the depthwise-separable convolution.
