# US 2019/0171930 A1 · Method and apparatus for processing convolution operation in neural network

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2019/0171930 A1
- Document ID: US20190171930A1
- Status: Published application
- Kind: A1
- Application number: 16/158,660
- Filed: 2018-10-12
- Published: 2019-06-06
- Earliest claimed: 2017-12-05
- Assignees: Samsung Electronics Co., Ltd.
- Applicants: Samsung Electronics Co., Ltd.
- Inventors: Sehwan LEE; Namjoon KIM; Joonho SONG; Junwoo JANG
- Web page: https://patentgrep.com/patent/US20190171930A1

## CPC

- G06N 3/063: using electronic means
- G06F 17/15: Correlation function computation {including computation of convolution operations (arithmetic circuits for sum of products per se, e.g. multiply-accumulators G06F7/5443; digital filters, e.g. FIR, IIR, adaptive filters H03H17/00)}
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/045: Combinations of networks
- G06T 1/20: Processor architectures; Processor configuration, e.g. pipelining
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

Provided are a method and apparatus for processing a convolution operation in a neural network, the method includes determining operands from input feature maps and kernels, on which a convolution operation is to be performed, dispatching operand pairs combined from the determined operands to multipliers in a convolution operator, generating outputs by performing addition and accumulation operations with respect to results of multiplication operations, and obtaining pixel values of output feature maps corresponding to a result of the convolution operation based on the generated outputs.

## Claims

### Claim 1

A method of processing a convolution operation in a neural network, the method comprising:
determining m first-bit feature map operands and n second-bit weight operands from input feature maps and kernels, respectively, on which the convolution operation is to be performed, wherein m and n are each a natural number;
dispatching m×n operand pairs of a feature map operand and a weight operand, which are combined from the first-bit feature map operands and the second-bit weight operands, respectively, to decomposed sub-multipliers in a convolution operator;
generating m×n outputs by performing addition and accumulation operations on results of multiplication operations performed by the decomposed sub-multipliers; and
obtaining pixel values of output feature maps corresponding to results of the convolution operation based on the m×n outputs.

### Claim 2 (depends on 1)

The method of claim 1, wherein the first-bit feature map operands and the second-bit weight operands are respectively determined based on pixel values of the input feature maps and weights of the kernels, which are processable in parallel.

### Claim 3 (depends on 2)

The method of claim 2, wherein the first-bit feature map operands are based on pixel values at different pixel locations in an input feature map from among the input feature maps.

### Claim 4 (depends on 3)

The method of claim 3, wherein the second-bit weight operands are based on weights at corresponding locations in different kernels from among the kernels, wherein the different kernels reference an input channel and different output channels of the input feature map.

### Claim 5 (depends on 3)

The method of claim 3, wherein the second-bit weight operands are based on weights at different locations in a kernel from among the kernels, wherein the kernel references an input channel and any one output channel of the input feature map.

### Claim 6 (depends on 2)

The method of claim 2, wherein the first-bit feature map operands are based on pixel values at corresponding pixel locations in different input feature maps from among the input feature maps, wherein the different input feature maps correspond to different input channels.

### Claim 7 (depends on 6)

The method of claim 6, wherein the second-bit weight operands are based on weights at corresponding locations in different kernels from among the kernels, wherein the different kernels correspond to the different input channels and any one output channel.

### Claim 8 (depends on 6)

The method of claim 6, wherein the second-bit weight operands are based on weights at corresponding locations in different kernels from among the kernels, wherein the different kernels correspond to the different input channels and different output channels.

### Claim 9 (depends on 1)

The method of claim 1, wherein:
the decomposed sub-multipliers respectively correspond to sub-logics of a k-bit multiplier, in response to the convolution operator comprising the k-bit multiplier having full precision of k bits,
the first-bit and the second-bit are each smaller than the k-bit, and
each of the decomposed sub-multipliers corresponds to a multiplier of the first-bit or a multiplier of the second-bit.

### Claim 10 (depends on 9)

The method of claim 9, wherein:
the first-bit feature map operands and the second-bit weight operands correspond to k/2-bit operands,
each of the decomposed sub-multipliers corresponds to a k/2-bit multiplier, and
the operand pairs, in which the first-bit feature map operands and the second-bit weight operands are mapped to each other, are respectively dispatched to the decomposed sub-multipliers.

### Claim 11 (depends on 9)

The method of claim 9, wherein:
each of the decomposed sub-multipliers corresponds to a k/2-bit multiplier,
operand pairs, in which the first-bit feature map operands and most significant bits of k/2 bits in the second-bit weight operands are mapped to each other, and operand pairs, in which the first-bit feature map operands and least significant bits of k/2 bits in the second-bit weight operands are mapped to each other, are respectively dispatched to the decomposed sub-multipliers, in response to the first-bit feature map operands being k/2-bit operands and the second-bit weight operands being k-bit operands, and
operand pairs, in which most significant bits of k/2 bits in the first-bit feature map operands and the second-bit weight operands are mapped to each other, and operand pairs, in which least significant bits of k/2 bits in the first-bit feature map operands and the second-bit weight operands are mapped to each other, are respectively dispatched to the decomposed sub-multipliers, in response to the first-bit feature map operands being k-bit operands and the second-bit weight operands being k/2-bit operands.

### Claim 12 (depends on 9)

The method of claim 9, wherein:
each of the decomposed sub-multipliers corresponds to a k/2-bit multiplier, and
operand pairs, in which most significant bits and least significant bits of k/2 bits in the first-bit feature map operands and most significant bits and least significant bits of k/2 bits in the second-bit weight operands are mapped to each other, are respectively dispatched to the decomposed sub-multipliers, in response to the first-bit feature map operands and the second-bit weight operands being k-bit operands.

### Claim 13 (depends on 1)

The method of claim 1, further comprising clock-gating a multiplication operation of a sub-multiplier to which a zero operand is dispatched, for zero skipping, in response to the zero operand being present in the m×n operand pairs.

### Claim 14 (depends on 1)

A non-transitory computer-readable storage medium storing instructions that, when executed by a processor, cause the processor to perform the image recognition method of claim 1.

### Claim 15

An apparatus for processing a convolution operation in a neural network, the apparatus comprising:
a processor is further configured to:
determine m first-bit feature map operands and n second-bit weight operands from input feature maps and kernels, respectively, on which the convolution operation is to be performed, wherein m and n are each a natural number;
dispatch m×n operand pairs of a feature map operand and a weight operand, which are combined from the first-bit feature map operands and second-bit weight operands, respectively to decomposed sub-multipliers in a convolution operator;
generate m×n outputs by performing addition and accumulation operations on results of multiplication operations performed by the decomposed sub-multipliers; and
obtain pixel values of output feature maps corresponding to results of the convolution operation based on the m×n outputs.

### Claim 16 (depends on 15)

The apparatus of claim 15, wherein the first-bit feature map operands and the second-bit weight operands are respectively determined based on pixel values of the input feature maps and weights of the kernels, which are processable in parallel.

### Claim 17 (depends on 16)

The apparatus of claim 16, wherein the first-bit feature map operands are based on pixel values at different pixel locations in an input feature map from among the input feature maps.

### Claim 18 (depends on 17)

The apparatus of claim 17, wherein the second-bit weight operands are based on weights at corresponding locations in different kernels from among the kernels, wherein the different kernels reference an input channel and different output channels of the input feature map, or
the second-bit weight operands are based on weights at different locations in one kernel from among the kernels, wherein the kernel references an input channel and any one output channel of the one input feature map.

### Claim 19 (depends on 16)

The apparatus of claim 16, wherein the first-bit feature map operands are based on pixel values at corresponding pixel locations in different input feature maps from among the input feature maps, wherein the different input feature maps correspond to different input channels.

### Claim 20 (depends on 19)

The apparatus of claim 19, wherein the second-bit weight operands are based on weights at corresponding locations in different kernels from among the kernels, wherein the different kernels correspond to the different input channels and any one output channel, or
the second-bit weight operands are based on weights at corresponding locations in different kernels from among the kernels, wherein the different kernels correspond to the different input channels and different output channels.

### Claim 21 (depends on 15)

The apparatus of claim 15, wherein:
the decomposed sub-multipliers respectively correspond to sub-logics of a k-bit multiplier, in response to the convolution operator comprising a k-bit multiplier having full precision of k bits,
the first-bit and the second-bit are each smaller than the k-bit, and
each of the decomposed sub-multipliers corresponds to a multiplier of the first-bit or a multiplier of the second-bit.

### Claim 22 (depends on 21)

The apparatus of claim 21, wherein:
the first-bit feature map operands and the second-bit weight operands correspond to k/2-bit operands,
each of the decomposed sub-multipliers corresponds to a k/2-bit multiplier, and
the operand pairs, in which the first-bit feature map operands and the second-bit weight operands are mapped to each other, are respectively dispatched to the decomposed sub-multipliers.

### Claim 23 (depends on 21)

The apparatus of claim 21, wherein:
each of the decomposed sub-multipliers corresponds to a k/2-bit multiplier,
operand pairs, in which the first-bit feature map operands and most significant bits of k/2 bits in the second-bit weight operands are mapped to each other, and operand pairs, in which the first-bit feature map operands and least significant bits of k/2 bits in the second-bit weight operands are mapped to each other, are respectively dispatched to the decomposed sub-multipliers, in response to the first-bit feature map operands being k/2-bit operands and the second-bit weight operands being k-bit operands, and
operand pairs, in which most significant bits of k/2 bits in the first-bit feature map operands and the second-bit weight operands are mapped to each other, and operand pairs, in which least significant bits of k/2 bits in the first-bit feature map operands and the second-bit weight operands are mapped to each other, are respectively dispatched to the decomposed sub-multipliers, in response to the first-bit feature map operands being k-bit operands and the second-bit weight operands being k/2-bit operands.

### Claim 24 (depends on 21)

The apparatus of claim 21, wherein:
each of the decomposed sub-multipliers corresponds to a k/2-bit multiplier, and
operand pairs, in which most significant bits and least significant bits of k/2 bits in the first-bit feature map operands and most significant bits and least significant bits of k/2 bits in the second-bit weight operands are mapped to each other, are respectively dispatched to the decomposed sub-multipliers, in response to the first-bit feature map operands and the second-bit weight operands are k-bit operands.

### Claim 25 (depends on 15)

The apparatus of claim 15, wherein the processor is further configured to, clock-gate a multiplication operation of a sub-multiplier to which a zero operand is dispatched, for zero skipping, in response to a zero operand being present in the m×n operand pairs.

### Claim 26 (depends on 15)

The apparatus of claim 15, further comprising a memory storing instructions that, when executed, configure the processor to determine the m first-bit feature map operands and the n second-bit weight operands, dispatch the m×n operand pairs, generate the m×n outputs, and obtain the output feature maps.