Back to results
US 12,524,656 B2Granted
Depthwise-convolution implementation on a neural processing core
AssigneeSamsung Electronics Co., Ltd.
InventorsJong Hoon Shin, Ali Shafiee Ardestani, Joseph H. Hassoun
Life of this patent
Estimating the term…
Abstract
A core of neural processing units is configured to efficiently process a depthwise convolution by maximizing spatial feature-map locality using adder trees. Data paths of activations and weights are inverted, and 2-to-1 multiplexers are every 2/9 multipliers along a row of multipliers. During a depthwise convolution operation, the core is operated using a RS×HW dataflow to maximize the locality of feature maps. For a normal convolution operation, the data paths of activations and weights may be configured for a normal convolution configuration and in which multiplexers are idle.
Claims18
A core of neural processing units (NPUs), comprising:
- a first memory that stores first values;
- a second memory that stores second values;
- and
- an array of NPUs arranged in C rows and K columns in which C and K comprise integers that are both divisible by 4, each NPU comprising 4×4 processing elements (PEs) arranged in 4 rows and 4 columns, each PE comprising a first input and a second input,
- in a first operational mode of the core of NPUs, the core of NPUs being configured as Int (C/3) row groups (RGs), each row group comprising:
- three consecutively adjacent rows of PEs,
- a first row RG1 comprising one or more 2 to 1 multiplexers having an output coupled to the first input of a PE11 designating the first row and a first column in each kth NPU for k=1 to K, the one or more 2 to 1 multiplexers comprising a first input and a second input,
- for k=1, the first input to the one or more 2 to 1 multiplexers being coupled to the first memory,
- for k=1 to K, the first input of each PE12 designating the first row and a second column and the first input of each PE14 designating the first row and a fourth column in each kth NPU being coupled to the second input of the one or more 2 to 1 multiplexers coupled to the PE11 in the kth NPU, to the first input of the one or more 2 to 1 multiplexers that is coupled to the PE11 in a (k+1)th NPU, and to the first memory,
- a second row RG2 comprising a 2 to 1 multiplexer having an output coupled to the first input of a PE23 designating the second row and a third column in each kth NPU for k=1 to K,
- for k=1, the first input of a PE21 designating the second row and the first column and a PE22 designating the second row and the second column in the kth NPU being coupled to the first input of the 2 to 1 multiplexer coupled to the PE23 in the kth NPU and to the first memory,
- for k=2 to K, the first input of the PE21 and the PE22 in each kth NPU being coupled to the second input of the 2 to 1 multiplexer coupled to the PE23 in the (k 1)th NPU, to the first input of a PE24 designating the second row and the fourth column in the (k 1)th NPU, to the first input of the 2 to 1 multiplexer coupled to the PE23 in the kth NiPU, and to the first memory,
- a third row RG3 comprising the first input of each of a PE31 through a PE34, wherein the PE31 designates a third row and the first column, and the PE34 designates the third row and the fourth column, the first input of each of the PE31 through the PE34 being coupled to the first input of each other PE31 through PE34 of the kth NPU, and to the first memory, wherein k=1 to K, and
- the second input of each PE in a column of PEs being coupled to the second memory.
11 more elements
A method to configure a core of neural processing units (NPUs), the core of NPUs comprising an array of NPUs arranged in C rows and K columns in which C and K comprise integers that are both divisible by 4, each NPU comprising 4×4 processing elements (PEs) arranged in 4 rows and 4 columns, each PE comprising a first input and a second input, the method comprising:
- for a first operational mode, configuring the core of NPUs to generate a depthwise convolution result from first values and second values by configuring the core of NPUs into Int (C/3) row groups (RGs), each row group comprising three consecutively adjacent rows of PEs;
- configuring a first row RG1 so that one or more 2 to 1 multiplexers has an output coupled to the first input of a PE11 designating the first row and a first column in each kth NPU for k=1 to K, the one or more 2 to 1 multiplexers comprising a first input and a second input,
- for k=1, configuring the first input to the one or more 2 to 1 multiplexers to be coupled to a first memory that stores the first values, and
- for k=1 to K, configuring the first input of each PE12 designating the first row and a second column and each PE14 designating the first row and a fourth column in each kth NPU to be coupled to the second input of the one or more 2 to 1 multiplexers that is coupled to the PE11 in the kth NPU, to be coupled to the first input of the one or more 2 to 1 multiplexers coupled to the PE11 in a (k+1)th NPU, and to be coupled to the first memory;
- configuring a second row RG2 so that a 2 to 1 multiplexer has an output coupled to the first input of a PE23 designating the second row and a third column in each kth NPU for k=1 to K,
- for k=1, configuring the first input of a PE21 designating the second row and the first column and the first input of a PE22 designating the second row and the second column in the kth NPU to be coupled to the first input of the 2 to 1 multiplexer coupled to the PE23 in the kth NPU and to be coupled to the first memory,
- for k=2 to K, configuring the first input of the PE21 and the PE22 in each kth NPU to be coupled to the second input of the 2 to 1 multiplexer coupled to the PE23 in the (k)th NPU, to be coupled to the first input of a PE24 designating the second row and the fourth column in the (k)th NPU, to be coupled to the first input of the 2 to 1 multiplexer coupled to the PE23 in the kth NPU, and to be coupled to the first memory,
- configuring a third row RG3 so that the first input of each of a PE31 through a PE34, the PE31 designating a third row and the first column, and the PE34 designating the third row and a fourth column, is coupled to the first input of each other PE31 through PE34 of the kth NPU, and is coupled to the first memory, wherein k=1 to K, and
- configuring the second input of each PE in a column of PEs to be coupled to a second memory that stores the second values.
6 more elements
Dependencies are inferred from the claim text.