# US 2020/0394516 A1 · Filter processing device and method of performing convolution operation at filter processing device

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2020/0394516 A1
- Document ID: US20200394516A1
- Status: Published application
- Kind: A1
- Application number: 16/896,565
- Filed: 2020-06-09
- Published: 2020-12-17
- Earliest claimed: 2019-06-17
- Assignees: Canon Kabushiki Kaisha
- Applicants: Canon Kabushiki Kaisha
- Inventors: Tsewei Chen; Masami Kato
- Web page: https://patentgrep.com/patent/US20200394516A1

## CPC

- G06F 7/5443: {Sum of products (for applications thereof, see the relevant places, e.g. G06F17/10, H03H17/00)}
- G06F 18/2413: based on distances to training or reference patterns
- G06F 18/2431: Multiple classes
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/0495: Quantised networks; Sparse networks; Compressed networks
- G06N 3/063: using electronic means
- G06N 3/08: Learning methods
- G06N 3/082: modifying the architecture, e.g. adding, deleting or silencing nodes or connections
- G06V 10/454: {Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]}
- G06V 10/764: using classification, e.g. of video objects
- G06V 10/82: using neural networks
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

There is provided with a filter processing device for performing a convolution operation using a filter on a feature plane. An acquisition unit acquires feature plane data, filter coefficients of the filter, and valid coefficient information which specifies filter coefficients to be used in the convolution operation among the filter coefficients of the filter. A calculation unit performs a convolution operation of the filter coefficients specified by the valid coefficient information and pieces of the feature plane data corresponding to the specified filter coefficients.

## Claims

### Claim 1

A filter processing device for performing a convolution operation using a filter on a feature plane, the filter processing device comprising:
an acquisition unit configured to acquire feature plane data, filter coefficients of the filter, and valid coefficient information which specifies filter coefficients to be used in the convolution operation among the filter coefficients of the filter; and
a calculation unit configured to perform a convolution operation of the filter coefficients specified by the valid coefficient information and pieces of the feature plane data corresponding to the specified filter coefficients.

### Claim 2 (depends on 1)

The filter processing device according to claim 1, wherein
the calculation unit is further configured to, in the convolution operation, omit a part of the convolution operation of a filter coefficient that is not specified by the valid coefficient information among filter coefficients of the filter and a piece of the feature plane data corresponding to the filter coefficient that is not specified by the valid coefficient information.

### Claim 3 (depends on 1)

The filter processing device according to claim 1, wherein
the acquisition unit is further configured to perform transfer of a piece of the feature plane data corresponding to a filter coefficient specified by the valid coefficient information to the calculation unit, and to omit transfer of a piece of the feature plane data corresponding to the filter coefficient that is not specified by the valid coefficient information to the calculation unit.

### Claim 4 (depends on 1)

The filter processing device according to claim 1, wherein
the acquisition unit is further configured to perform transfer of a filter coefficient specified by the valid coefficient information to the calculation unit, and to omit transfer of a filter coefficient that is not specified by the valid coefficient information to the calculation unit.

### Claim 5 (depends on 1)

The filter processing device according to claim 1, wherein
the valid coefficient information is associated with the filter.

### Claim 6 (depends on 5)

The filter processing device according to claim 5, wherein
the valid coefficient information is common for at least two filters.

### Claim 7 (depends on 6)

The filter processing device according to claim 6, wherein
the calculation unit is further configured to perform a convolution operation using filter coefficients of a first filter specified by a first valid coefficient information, and subsequently perform a convolution operation using filter coefficients of a second filter specified by the first valid coefficient information.

### Claim 8 (depends on 1)

The filter processing device according to claim 1, wherein
a plurality of filters classified into a plurality of groups are used by the filter processing device,
the valid coefficient information is associated with the group, and
the acquisition unit is further configured to acquire the valid coefficient information associated with a group to which the filter belongs.

### Claim 9 (depends on 1)

The filter processing device according to claim 1, wherein
the filter processing device is configured to perform processing according to a neural network including a plurality of layers, and
the filter processing device is further configured to calculate each of a plurality of feature planes of a second layer by performing filter processing on a plurality of feature planes of a first layer using filters respectively corresponding to the plurality of feature planes of the first layer.

### Claim 10 (depends on 9)

The filter processing device according to claim 9, wherein
the valid coefficient information is common for the filters respectively corresponding to the plurality of feature planes of the first layer.

### Claim 11 (depends on 10)

The filter processing device according to claim 10, wherein
the filter processing device is further configured to calculate each of a plurality of feature planes of a fourth layer by performing filter processing on a plurality of feature planes of a third layer using filters respectively corresponding to the plurality of feature planes of the third layer,
the valid coefficient information is common for the filters respectively corresponding to the plurality of feature plane of the third layer, and
the valid coefficient information is different between the filters respectively corresponding to the plurality of feature planes of the first layer and the filters respectively corresponding to the plurality of feature planes of the third layer.

### Claim 12 (depends on 9)

The filter processing device according to claim 9, wherein
in terms of the filters respectively corresponding to the plurality of feature planes of the first layer, the valid coefficient information for the filter is different for each feature plane.

### Claim 13 (depends on 1)

The filter processing device according to claim 1, further comprising:
a coefficient holding unit configured to store the filter coefficients, wherein
the acquisition unit is further configured to buffer a part of filter coefficients stored in the coefficient holding unit.

### Claim 14 (depends on 13)

The filter processing device according to claim 13, wherein
the acquisition unit is further configured to buffer a filter coefficient specified by the valid coefficient information, among the filter coefficients stored in the coefficient holding unit.

### Claim 15 (depends on 1)

The filter processing device according to claim 1, further comprising:
a feature plane holding unit configured to store the feature plane data, wherein
the acquisition unit is further configured to buffer a part of the feature plane data stored in the feature plane holding unit.

### Claim 16 (depends on 15)

The filter processing device according to claim 15, wherein
the acquisition unit is further configured to buffer a piece of the feature plane data corresponding to a filter coefficient specified by the valid coefficient information, among the feature plane data stored in the feature plane holding unit.

### Claim 17 (depends on 1)

The filter processing device according to claim 1, wherein
the valid coefficient information specifies, for each position of the filter, whether to use a filter coefficient in the convolution operation.

### Claim 18 (depends on 1)

The filter processing device according to claim 1, wherein
the valid coefficient information includes information indicating a position of one filter coefficient in the filter to be used in the convolution operation, and information indicating a relative position of another filter coefficient to be used in the convolution operation with respect to the one filter coefficient.

### Claim 19

A method of performing, at a filter processing device, a convolution operation using a filter on a feature plane, the method comprising:
acquiring feature plane data, filter coefficients of the filter, and valid coefficient information which specifies filter coefficients to be used in the convolution operation among the filter coefficients of the filter; and
performing a convolution operation of the filter coefficients specified by the valid coefficient information and pieces of the feature plane data corresponding to the specified filter coefficients.
