# US 2024/0004615 A1 · Convolution operation method and apparatus, matrix decompression device, and graphics processor

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2024/0004615 A1
- Document ID: US20240004615A1
- Status: Published application
- Kind: A1
- Application number: 18/216,809
- Filed: 2023-06-30
- Published: 2024-01-04
- Earliest claimed: 2022-07-01
- Applicants: Glenfly Tech Co., Ltd.
- Inventors: Zhongyu TAO; Huaisheng ZHANG; Renyu BIAN
- Web page: https://patentgrep.com/patent/US20240004615A1

## CPC

- G06F 7/78: for changing the order of data flow, e.g. matrix transposition or LIFO buffers; Overflow or underflow handling therefor
- G06F 17/15: Correlation function computation {including computation of convolution operations (arithmetic circuits for sum of products per se, e.g. multiply-accumulators G06F7/5443; digital filters, e.g. FIR, IIR, adaptive filters H03H17/00)}
- Y02D 10/00: Energy efficient computing, e.g. low power processors, power management or thermal management
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- Y02D: CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE

## Abstract

Convolution operation method and apparatus, matrix decompression device and graphics processor are provided. The method includes: loading, from a preset memory layout, at least one target feature tile constituting any sub-feature map in an original feature map for the any sub-feature map; the memory layout being obtained by writing at least one feature tile into memory according to preset way of data arrangement; the at least one feature tile being obtained by tiling the original feature map; decompressing a feature map which is composed of the at least one target feature tile according to a convolution parameter of a convolutional layer to obtain a destination decompressed matrix; performing a matrix multiplication operation on the destination decompressed matrix and the decompressed matrix corresponding to a convolution kernel to obtain a convolution operation result of the original feature map. The present disclosure may improve the convolution operation efficiency.

## Claims

### Claim 1

A convolution operation method, comprising:
loading at least one target feature tile, which constitutes any one of sub-feature maps in an original feature map, from a preset memory layout for the any one of the sub-feature maps; wherein the memory layout is obtained by writing at least one feature tile into a memory according to a preset way of data arrangement, and the at least one feature tile is obtained by tiling the original feature map;
decompressing a feature map which is composed of the at least one target feature tile according to a convolution parameter of a convolutional layer to obtain a destination decompressed matrix; and
performing a matrix multiplication operation on the destination decompressed matrix and a decompressed matrix corresponding to a convolution kernel and obtaining a convolution operation result of the original feature map.

### Claim 2 (depends on 1)

The convolution operation method of claim 1, further comprising:
tiling the original feature map to obtain the at least one feature tile; and
according to the way of data arrangement, writing each feature tile sequentially into the memory in order to obtain the memory layout; wherein an arrangement dimension of the way of data arrangement comprises at least a batch processing dimension, a channel dimension and a position dimension of each feature tile in the original feature map.

### Claim 3 (depends on 2)

The convolution operation method of claim 2, wherein the according to the way of data arrangement, writing each feature tile sequentially into the memory in order to obtain the memory layout comprises:
writing at least one feature tile of a same target position in the original feature map into the memory sequentially along a direction which corresponds to the channel dimension in order to obtain a feature tile brick corresponding to the target position.

### Claim 4 (depends on 2)

The convolution operation method of claim 2, wherein the tiling the original feature map to obtain the at least one feature tile comprises:
obtaining a tile sample plate which is used to tile the original feature map;
determining a size of the tile sample plate in at least one direction;
performing zero padding on the original feature map to enable a size of the zero-padded feature map in a direction to be a multiple of a size of the tile sample plate in the direction; and
according to the tile sample plate, tiling the zero-padded feature map to obtain the at least one feature tile.

### Claim 5 (depends on 1)

The convolution operation method of claim 1, wherein there exist, in the memory layout, tile index coordinates corresponding to each feature tile; and
wherein the loading at least one target feature tile, which constitutes any one of sub-feature maps in an original feature map, from a preset memory layout for the any one of the sub-feature maps comprises:
obtaining decompressed matrix position coordinates corresponding to any one of the sub-feature maps; the decompressed matrix position coordinates being used to represent position information of the destination decompressed matrix in a decompressed matrix corresponding to the original feature map;
mapping the decompressed matrix position coordinates to target tile index coordinates; the target tile index coordinates being tile index coordinates, in the memory layout, corresponding to at least one target feature tile which constitutes any one of the sub-feature maps; and
loading a feature tile corresponding to the target tile index coordinates in the memory layout to obtain a target feature tile.

### Claim 6 (depends on 1)

The convolution operation method of claim 1, wherein the decompressing the feature map which is composed of the at least one target feature tile according to the convolution parameter of the convolutional layer to obtain the destination decompressed matrix comprises:
decompressing the feature map, which is composed of the at least one target feature tile, according to a convolution parameter of a convolutional layer to obtain a decompressed matrix;
performing a transpose operation on the decompressed matrix to obtain the destination decompressed matrix.

### Claim 7 (depends on 1)

The convolution operation method of claim 1, further comprising:
obtaining a convolutional layer to which a current convolution operation belongs;
parsing a convolution pattern of the convolutional layer to determine a convolution parameter of the convolutional layer.

### Claim 8

A convolution operation apparatus, comprising:
a reading module, which is configured to read an original feature map used for a convolution operation;
a loading module, which is configured to load at least one target feature tile which constitutes any one of sub-feature maps from a preset memory layout for the any one of the sub-feature maps in an original feature map; wherein the memory layout is obtained by writing at least one feature tile into a memory according to a preset way of data arrangement, the at least one feature tile is obtained by tiling the original feature map; a way of memory layout includes at least a batch processing dimension, a channel dimension and a position dimension of each feature tile in the original feature map;
a decompression module, which is configured to decompress a feature map which is composed of the at least one target feature tile according to a convolution parameter of a convolutional layer to obtain a destination decompressed matrix; and
an operation module, which is configured to perform a matrix multiplication operation on the destination decompressed matrix and a decompressed matrix corresponding to a convolution kernel to obtain a convolution operation result for the original feature map.

### Claim 9 (depends on 8)

The convolution operation apparatus of claim 8, wherein the convolution operation apparatus is further configured to:
tile the original feature map to obtain at least one feature tile; and
write each feature tile sequentially to the memory according to the way of data arrangement, to obtain the memory layout;
wherein, an arrangement dimension of the way of data arrangement comprises at least a batch processing dimension, a channel dimension and a position dimension of each feature tile in the original feature map.

### Claim 10 (depends on 9)

The convolution operation apparatus of claim 9, wherein the convolution operation apparatus is further configured to write at least one feature tile of a same target position in the original feature map into the memory sequentially along a direction which corresponds to the channel dimension in order to obtain a feature tile brick corresponding to the target position.

### Claim 11 (depends on 9)

The convolution operation apparatus of claim 9, wherein the convolution operation apparatus is further configured to:
obtain a tile sample plate which is used to tile the original feature map;
determine a size of the tile sample plate in at least one direction;
perform zero padding on the original feature map to enable a size of the zero-padded feature map in a direction to be a multiple of a size of the tile sample plate in the direction; and
tile the zero-padded feature map according to the tile sample plate to obtain the at least one feature tile.

### Claim 12 (depends on 8)

The convolution operation apparatus of claim 8, wherein there exist, in the memory layout, tile index coordinates corresponding to each feature tile; and
wherein the loading module is configured to:
obtain decompressed matrix position coordinates corresponding to any one of the sub-feature maps; wherein the decompressed matrix position coordinates are used to represent a position information of the destination decompressed matrix in a decompressed matrix corresponding to the original feature map;
map the decompressed matrix position coordinates to target tile index coordinates; wherein the target tile index coordinates are tile index coordinates, in the memory layout, corresponding to at least one target feature tile which constitutes any one of the sub-feature maps; and
load a feature tile corresponding to the target tile index coordinates in the memory layout to obtain a target feature tile.

### Claim 13 (depends on 8)

The convolution operation apparatus of claim 8, wherein the decompression module is configured to: decompress the feature map which is composed of the at least one target feature tile, to obtain a decompressed matrix according to convolution parameters of a convolutional layer; and perform a transpose operation on the decompressed matrix to obtain a destination decompressed matrix.

### Claim 14 (depends on 8)

The convolution operation apparatus of claim 8, wherein the convolution operation apparatus is further configured to: obtain a convolutional layer to which a current convolution operation belongs; and parse a convolution pattern of the convolutional layer to determine a convolution parameter of the convolutional layer.

### Claim 15

A matrix decompression device, comprising: a tile collector, a pattern parser, a matrix processing module and a matrix buffer, wherein:
the tile collector is configured to obtain at least one target feature tile, which constitutes any one of sub-feature maps in an original feature map, from a texture unit; the at least one target feature tile is loaded by the texture unit from a preset memory layout;
the pattern parser is configured to obtain a convolution parameter of a convolutional layer;
the matrix processing module is configured to perform a decompression processing on a feature map, which is composed of the at least one target feature tile, according to the convolution parameter to obtain a destination decompressed matrix; and
the matrix buffer is configured to cache the destination decompressed matrix based on which an execute unit is able to generate a convolution operation result of the original feature map.

### Claim 16 (depends on 15)

The matrix decompression device of claim 15, the matrix processing module comprises a matrix decompression engine and a matrix transpose control, wherein:
the matrix decompression engine is configured to decompress the feature map, which is composed of at least one target feature tile, according to the convolution parameter to obtain a decompressed matrix;
the matrix transpose control is configured to perform a transpose operation on the decompressed matrix to obtain the destination decompressed matrix.

### Claim 17 (depends on 16)

The matrix decompression device of claim 16, wherein the convolution parameter comprises a convolution stride and a convolution kernel size; the matrix decompression engine is configured to convert, according to the convolution step size and the convolution kernel size, a feature map which is composed of the at least one feature tile into at least one row vector based on a position in the original map in sequence, and to splice the at least one row vector into a feature map matrix to obtain the decompressed matrix.

### Claim 18 (depends on 15)

The matrix decompression device of claim 15, wherein the pattern parser is configured to: obtain a current convolutional layer to which a convolution operation belongs; and parse a convolution pattern of the current convolutional layer and determine a convolution parameter of the convolutional layer.

### Claim 19 (depends on 15)

The matrix decompression device of claim 15, wherein the matrix buffer is further configured to transmit the destination decompressed matrix to a high-speed shared memory of the execute unit.
