# US 11,113,361 B2 · Electronic apparatus and control method thereof

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,113,361 B2
- Document ID: US11113361B2
- Status: Granted
- Kind: B2
- Application number: 16/295,599
- Filed: 2019-03-07
- Published: 2019-09-12
- Granted: 2021-09-07
- Earliest claimed: 2018-03-07
- Assignees: Samsung Electronics Co., Ltd.
- Applicants: Samsung Electronics Co., Ltd.
- Inventors: Younghwan Park; Kyounghoon Kim; Dongkwan Suh; Hansu Cho; Keshava Prasad Nagaraja; Sukjin Kim; Hyunjung Kim
- Web page: https://patentgrep.com/patent/US11113361B2

## CPC

- G06F 17/153: {Multidimensional correlation or convolution}
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/0495: Quantised networks; Sparse networks; Compressed networks
- G06N 3/063: using electronic means
- G06N 3/084: Backpropagation, e.g. using gradient descent
- H03K 19/20: characterised by logic function, e.g. AND, OR, NOR, NOT circuits (H03K19/003 - H03K19/01 take precedence)
- H03M 7/30: Compression (speech analysis-synthesis for redundancy reduction G10L19/00; for image communication H04N); Expansion; Suppression of unnecessary data, e.g. redundancy reduction
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- H03K: PULSE TECHNIQUE (measuring pulse characteristics G01R; modulating sinusoidal oscillations with pulses H03C; transmission of digital information H04L; discriminator circuits detecting phase difference between two signals by counting or integrating cycles of oscillation H03D3/04; automatic control, starting, synchronisation or stabilisation of generators of electronic oscillations or pulses where the type of generator is irrelevant or unspecified H03L; coding, decoding or code conversion, in general H03M)
- H03M: CODING; DECODING; CODE CONVERSION IN GENERAL (using fluidic means F15C4/00; optical analogue/digital converters G02F7/00; coding, decoding or code conversion, specially adapted for particular applications, see the relevant subclasses, e.g. G01D, G01R, G06F, G06T, G09G, G10L, G11B, G11C, H04B, H04L, H04M, H04N; ciphering or deciphering for cryptography or other purposes involving the need for secrecy G09C)

## Abstract

An electronic apparatus is provided. The electronic apparatus includes a storage storing object data and kernel data, and a processor including a plurality of processing elements arranged in a matrix formation, wherein the processor is configured to input corresponding first elements among a plurality of first elements included in the object data into processing elements arranged in a first row among the plurality of processing elements, and input a plurality of second elements included in the kernel data sequentially into the processing elements arranged in the first row to perform operations between the corresponding first elements and the plurality of second elements, to identify a depth in which a first element and a second element have a non-zero value, and to input the first element and the second element corresponding to the identified depth into a calculator included in each of the processing elements arranged in the first row to perform a convolution operation.

## Claims

### Claim 1

An electronic apparatus, comprising:
a storage configured to store object data and kernel data; and
a processor comprising a plurality of processing elements arranged in a matrix formation,
wherein the processor is configured to:
input corresponding first elements, among a plurality of first elements included in the object data, respectively into processing elements arranged in a first row among the plurality of processing elements,
input a plurality of second elements, included in the kernel data, sequentially into the processing elements arranged in the first row to perform operations between the corresponding first elements and the plurality of second elements,
compare, through a logic circuit included in each of the processing elements arranged in the first row, the corresponding first elements and the plurality of second elements input to the processing elements arranged in the first row by depth, the corresponding first elements and the plurality of second elements are composed of an equal number of plural depths,
identify a depth among the plural depths in which a first element and a second element, among the compared corresponding first elements and the plurality of second elements input to the processing elements arranged in the first row, have a non-zero value, and
input the first element and the second element corresponding to the identified depth into a calculator included in each of the processing elements arranged in the first row to perform a convolution operation,
wherein the processor is further configured to:
based on operations with respect to an input second element, among the plurality of second elements, being completed in the processing elements arranged in the first row, input another second element, among the plurality of second elements, into the processing elements arranged in the first row;
based on operations with respect to the input plurality of second elements being completed in the processing elements arranged in the first row, compare, through a logic circuit included in each of processing elements arranged in a second row, a corresponding first elements, among the plurality of first elements, and the plurality of second elements respectively input to the processing elements arranged in the second row by depth;
identify a depth in which a first element and a second element, among the compared corresponding first elements and the plurality of second elements input to the processing elements arranged in the second row, have a non-zero value; and
input the first element and the second element corresponding to the identified depth, among the compared corresponding first elements and the plurality of second elements input to the processing elements arranged in the second row, into a calculator included in each of the processing elements arranged in the second row to perform a convolution operation.

### Claim 2 (depends on 1)

The electronic apparatus as claimed in claim 1, wherein:
the processor further comprises a controller configured to control the plurality of processing elements.

### Claim 3 (depends on 2)

The electronic apparatus as claimed in claim 2, wherein the plurality of processing elements comprise:
a first buffer configured to store first elements input to the plurality of processing elements and a first bitmap indicating whether the input first elements are zero or non-zero by depth; and
a second buffer configured to store second elements input to the plurality of processing elements and a second bitmap indicating whether the input second element are zero or non-zero by depth, and
wherein the controller is further configured to compare, through the logic circuit included in each of the plurality of processing elements, the first bitmap and the second bitmap, and to identify a depth in which the input first elements and the input second elements have a non-zero value.

### Claim 4 (depends on 3)

The electronic apparatus as claimed in claim 3, wherein the controller is further configured to compare, through the logic circuit included in each of the plurality of processing elements, positions of “1” on respective arrays of the first bitmap and the second bitmap, and based on “1” being at a same position of the respective arrays of the first bitmap and the second bitmap, identify a depth corresponding to the same position of the input first elements and the input second elements.

### Claim 5 (depends on 4)

The electronic apparatus as claimed in claim 4, wherein:
the logic circuit included in each of the plurality of processing elements comprises a plurality of logic gates equal to or greater than a bit number of the first bitmap and the second bitmap; and
the controller is further configured to input a value in a same position on the respective arrays of the first bitmap and the second bitmap into a logic gate, and to compare the positions of “1” on the respective arrays of the first bitmap and the second bitmap.

### Claim 6 (depends on 5)

The electronic apparatus as claimed in claim 5, wherein:
the logic gate is a gate for performing an AND operation; and
the controller is further configured to identify a depth corresponding to a position on an array in which a resultant value of the AND operation of the logic gate in the input first elements and the input second elements is “1”.

### Claim 7 (depends on 1)

The electronic apparatus as claimed in claim 1, wherein each of the processing elements arranged in the first row is configured to shift an operation value obtained by multiplying a first non-zero element by a second non-zero element, both of which have a same depth in the input corresponding first elements and the input plurality of second elements, to an adjacent processing element in a row direction, and to perform accumulation on operation values between adjacent processing elements.

### Claim 8 (depends on 1)

The electronic apparatus as claimed in claim 1, wherein:
the processor further comprises a plurality of controllers for individually controlling the plurality of processing elements for each column; and
each of the plurality of controllers is configured to, based on operations by the input plurality of second elements being completed in a processing element in one column among the processing elements arranged in the first row, input another second element among the plurality of second elements into a processing element in which the operations are completed.

### Claim 9 (depends on 8)

The electronic apparatus as claimed in claim 8, wherein:
the input corresponding first elements and the input plurality of second elements are data compressed in a depth direction by removing a zero element; and
each of the plurality of controllers is further configured to, based on a first bitmap indicating whether a first element before compression is zero or non-zero by depth, and a second bitmap indicating whether a second element before compression is zero or non-zero by depth, identify a depth in which the first element and the second element before compression have a non-zero value, identify a first element and a second element corresponding to the identified depth from the compressed data, and input the identified first element and the identified second element into the calculator included in each of the processing elements arranged in the first row to perform the convolution operation.

### Claim 10

A controlling method for an electronic apparatus comprising a storage for storing object data and kernel data, and a processor including a plurality of processing elements arranged in a matrix formation, the controlling method comprising:
inputting corresponding first elements, among a plurality of first elements included in the object data, respectively to processing elements arranged in a first row among the plurality of processing elements;
inputting a plurality of second elements, included in the kernel data, sequentially into the processing elements arranged in the first row;
comparing, through a logic circuit included in each of the processing elements arranged in the first row, the corresponding first elements with the plurality of second elements input to the processing elements arranged in the first row by depth, the corresponding first elements and the plurality of second elements are composed of an equal number of plural depths;
identifying a depth among the plural depths in which a first element and a second element, among the compared corresponding first elements and the plurality of second elements input to the processing elements arranged in the first row, have a non-zero value;
inputting the first element and the second element corresponding to the identified depth into a calculator included in each of the processing elements arranged in the first row to perform a convolution operation;
based on operations with respect to an input second element, among the plurality of second elements, being completed in the processing elements arranged in the first row, inputting another second element, among the plurality of second elements, into the processing elements arranged in the first row;
comparing, through a logic circuit included in each of processing elements arranged in a second row, corresponding first elements, among the plurality of first elements, and the plurality of second elements respectively input to the processing elements arranged in the second row by depth;
identifying a depth in which a first element and a second element, among the compared corresponding first elements and the plurality of second elements input to the processing elements arranged in the second row, have a non-zero value; and
inputting the first element and the second element corresponding to the identified depth, among the compared corresponding first elements and the plurality of second elements input to the processing elements arranged in the second row, into a calculator included in each of the processing elements arranged in the second row to perform a convolution operation.

### Claim 11 (depends on 10)

The method as claimed in claim 10, wherein the plurality of processing elements comprise:
a first buffer configured to store first elements into to the plurality of processing elements and a first bitmap indicating whether the input first elements are zero or non-zero by depth; and
a second buffer configured to store second elements input to the plurality of processing elements, and a second bitmap indicating whether the input second elements are zero or non-zero by depth,
wherein the identifying comprises comparing, through the logic circuit included in each of the plurality of processing elements, the first bitmap with the second bitmap, and identifying a depth in which the input first elements and the input second elements have a non-zero value.

### Claim 12 (depends on 11)

The method as claimed in claim 11, wherein the identifying further comprises comparing, through the logic circuit included in each of the plurality of processing elements, positions of “1” on respective arrays of the first bitmap and the second bitmap, and based on “1” being at a same position of the respective arrays of the first bitmap and the second bitmap, identifying a depth corresponding to the same position of the input first elements and the input second elements.

### Claim 13 (depends on 12)

The method as claimed in claim 12, wherein:
the logic circuit included in each of the plurality of processing elements comprises a plurality of logic gates equal to or greater than a bit number of the first bitmap and the second bitmap; and
the identifying further comprises inputting a value in a same position on the respective arrays of the first bitmap and the second bitmap into a logic gate, and comparing the positions of “1” on the respective arrays of the first bitmap and the second bitmap.

### Claim 14 (depends on 13)

The method as claimed in claim 13, wherein:
the logic gate is a gate to perform an AND operation; and
the identifying further comprises identifying a depth corresponding to a position on an array in which a resultant value of the AND operation of the logic gate in the input first elements and the input second elements is “1”.

### Claim 15 (depends on 10)

The method as claimed in claim 10, wherein the performing the convolution operation comprises shifting an operation value obtained by multiplying a first non-zero element by a second non-zero element, both of which have a same depth in the input corresponding first elements and the input plurality of second elements, to an adjacent processing element in a row direction, and performing accumulation on operation values between adjacent processing elements.

### Claim 16 (depends on 10)

The method as claimed in claim 10, further comprising based on operations by the input plurality of second elements being completed in a processing element in one column among the processing elements arranged in the first row, inputting another second element among the plurality of second elements into a processing element in which the operations are completed.

### Claim 17 (depends on 16)

The method as claimed in claim 16, wherein:
the input corresponding first elements and the input plurality of second elements are data compressed in a depth direction by removing a zero element; and
the identifying comprises, based on a first bitmap indicating whether a first element before compression is zero or non-zero by depth, and a second bitmap indicating whether a second element before compression is zero or non-zero by depth, identifying a depth in which the first element and the second element before compression have a non-zero value, and identifying a first element and a second element corresponding to the identified depth from the compressed data.