# US 2021/0334603 A1 · Dividing pattern determination device capable of reducing amount of computation, dividing pattern determination method, learning device, learning method, and storage medium

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0334603 A1
- Document ID: US20210334603A1
- Status: Published application
- Kind: A1
- Application number: 17/234,609
- Filed: 2021-04-19
- Published: 2021-10-28
- Earliest claimed: 2020-04-28
- Assignees: Canon Kabushiki Kaisha
- Applicants: Canon Kabushiki Kaisha
- Inventors: Dai Miyauchi
- Web page: https://patentgrep.com/patent/US20210334603A1

## CPC

- H04N 19/119: Adaptive subdivision aspects, e.g. subdivision of a picture into rectangular or non-rectangular coding blocks
- G06F 18/2193: {based on specific statistical tests}
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06N 3/09: Supervised learning
- G06V 10/22: by selection of a specific region containing or referencing a pattern; Locating or processing of specific regions to guide the detection or recognition
- G06V 10/40: Extraction of image or video features
- G06V 10/7715: {Feature extraction, e.g. by transforming the feature space, e.g. multi-dimensional scaling [MDS]; Mappings, e.g. subspace methods}
- G06V 10/7796: {based on specific statistical tests}
- H04N 19/14: Coding unit complexity, e.g. amount of activity or edge presence estimation (H04N19/146 takes precedence)
- H04N 19/172: the region being a picture, frame or field
- H04N 19/176: the region being a block, e.g. a macroblock
- H04N 19/30: using hierarchical techniques, e.g. scalability (H04N19/63 takes precedence)
- H04N 19/85: using pre-processing or post-processing specially adapted for video compression
- H04N 19/96: Tree coding, e.g. quad-tree coding
- G06N 20/10: using kernel methods, e.g. support vector machines [SVM]
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

A dividing pattern determination device capable of reducing the amount of computation performed when determining a dividing pattern of an image. An image for which a dividing pattern is expressed by a hierarchical structure for each predetermined area is input to a feature extraction section, and the feature extraction section generates, based on the input image, for the predetermined area, a hierarchy map in which a value indicative of a block size is associated with each of a plurality of blocks in the predetermined area. A determination section determines a dividing pattern of the image based on the generated hierarchy map.

## Claims

### Claim 1

A device comprising:
at least one processor; and
a memory coupled to the at least one processor, the memory having instructions that, when executed by the processor, perform the operations as:
a generation unit configured to have an image fir which a dividing pattern is expressed by a hierarchical structure for each predetermined area, input thereto, and generate, based on the input image, for the predetermined area, a hierarchy map in which a value indicative of a block size is associated with each of a plurality of blocks in the predetermined area; and
a determination unit configured to determine a dividing pattern of the image based on the generated hierarchy map.

### Claim 2 (depends on 1)

The device according to claim 1, wherein the generation unit generates the hierarchy map using a learning model subjected to machine learning.

### Claim 3 (depends on 1)

The device according to claim 1, further comprising a correction unit configured to, in a case where, in the hierarchy map, there is a block in which inconsistency is generated between layers of the hierarchical structure, correct the inconsistency based on the block and a plurality of blocks around the block.

### Claim 4 (depends on 3)

The device according to claim 3, wherein the correction unit performs correction, with respect to blocks shallower in layer of the hierarchical structure than a predetermined layer, for deepening a layer where blocks are to be divided.

### Claim 5 (depends on 2)

The device according to claim 2, wherein the learning model is a neural network, and
wherein the neural network is subjected to machine learning using back propagation.

### Claim 6 (depends on 5)

The device according to claim 5, wherein the neural network includes a plurality of convolution sections and a plurality of pooling sections, and the convolution sections each convolute pixels in the predetermined area with pixels around the predetermined area.

### Claim 7 (depends on 5)

The device according to claim 5, wherein in a case where the neural network outputs a floating-point number, the generation unit converts the floating-point number to an integer value indicative of a size of the block.

### Claim 8 (depends on 1)

The device according to claim 1, wherein the image is one of images forming a moving image which is encoded based on HEVC standard.

### Claim 9

A device comprising:
at least one processor; and
a memory coupled to the at least one processor, the memory having instructions that, when executed by the processor, perform the operations as:
a generation unit configured to have an image for which a dividing pattern is expressed by a hierarchical structure for each predetermined area, input thereto, and generate, based on a feature of the input image, for the predetermined area, a feature map indicating whether or not required to be divided, with respect to each of a plurality of blocks in the predetermined area, for each of layers of the hierarchical structure; and
a determination unit configured to determine a dividing pattern of the image based on a plurality of the generated feature maps.

### Claim 10 (depends on 9)

The device according to claim 9, wherein the generation unit generates the feature map using a learning model subjected to machine learning.

### Claim 11 (depends on 9)

The device according to claim 9, further comprising a correction unit configured to, in a case where a block in an upper layer indicates not to be divided and a corresponding block in a lower layer indicates to be divided, perform correction such that the block in the upper layer indicates to be divided.

### Claim 12 (depends on 9)

The device according to claim 9, wherein higher priority is given to determination of whether or not required to divide blocks in a feature map of an upper layer than determination of whether or not required to divide blocks in a feature map of a lower layer.

### Claim 13 (depends on 12)

The device according to claim 12, wherein in a case where a block in the upper layer indicates not to be divided, a plurality of blocks in a lower layer corresponding to the block in the upper layer are excluded from an object to be processed.

### Claim 14 (depends on 13)

The device according to claim 13, wherein the generation unit generates a data string for the predetermined area, from which a block excluded from the object to be processed is omitted, for each layer of the hierarchical structure.

### Claim 15 (depends on 9)

The device according to claim 9, wherein the learning model is subjected to machine learning such that a value of a weight assigned to a mean squared error is set to be higher as a layer of the hierarchical structure is deeper.

### Claim 16 (depends on 9)

The device according to claim 9, wherein the learning model is a neural network, and
wherein the neural network is subjected to machine learning using hack propagation.

### Claim 17 (depends on 16)

The device according to claim 16, wherein the neural network includes a plurality of convolution sections and a plurality of pooling sections, and the convolution sections each convolute pixels in the predetermined area with pixels around the predetermined area.

### Claim 18 (depends on 16)

The device according to claim 16, wherein in a case where the neural network outputs a floating-point number, the generation unit converts the floating-point number to an integer value indicative of a size of the block.

### Claim 19 (depends on 9)

The device according to claim 9, wherein the image is one of images forming a moving image which is encoded based on HEVC standard.

### Claim 20

A device comprising:
at least one processor; and
a memory coupled to the at least one processor, the memory having instructions that, when executed by the processor, perform the operations as:
a generation unit configured to have an image for learning for which a dividing pattern is expressed by a hierarchical structure for each predetermined area, input thereto, and generate, based on the input image, for the predetermined area, a hierarchy map in which a value indicative of a block size is associated with each of a plurality of blocks in the predetermined area,
wherein the learning device performs machine learning such that a mean squared error between teacher data associated with the image for learning and the hierarch map is reduced.

### Claim 21

A method comprising:
having an image for which a dividing pattern is expressed by a hierarchical structure for each predetermined area, input thereto, and generating, based on the input image, for the predetermined area, a hierarchy map in which a value indicative of a block size is associated with each of a plurality of blocks in the predetermined area; and
determining a dividing pattern of the image based on the generated hierarchy map.

### Claim 22

A method comprising:
having an image for which a dividing pattern is expressed by a hierarchical structure for each predetermined area, input thereto, and generating, based on a feature of the input image, for the predetermined area, a feature map indicating whether or not required to be divided, with respect to each of a plurality of blocks in the predetermined area, for each of layers of the hierarchical structure; and
determining a dividing pattern of the image based on a plurality of the generated feature maps.

### Claim 23

A method comprising:
having an image for learning for which a dividing pattern is expressed by a hierarchical structure for each predetermined area, input thereto, and generating, based on the input image, for the predetermined area, a hierarchy map in which a value indicative of a block size is associated with each of a plurality of blocks in the predetermined area; and
performing machine learning such that a mean squared error between teacher data associated with the image for learning and the hierarch map is reduced.

### Claim 24

A non-transitory computer-readable storage medium storing a program for causing a computer to execute a method, the method comprising:
having an image for which a dividing pattern is expressed by a hierarchical structure for each predetermined area, input thereto, and generating, based on the input image, for the predetermined area, a hierarchy map in which a value indicative of a block size is associated with each of a plurality of blocks in the predetermined area; and
determining a dividing pattern of the image based on the generated hierarchy map.

### Claim 25

A non-transitory computer-readable storage medium storing a program for causing a computer to execute a method, the method comprising:
having an image for which a dividing pattern is expressed by a hierarchical structure for each predetermined area, input thereto, and generating, based on a feature of the input image, for the predetermined area, a feature map indicating whether or not required to be divided, with respect to each of a plurality of blocks in the predetermined area, for each of layers of the hierarchical structure; and
determining a dividing pattern of the image based on a plurality of the generated feature maps.

### Claim 26

A non-transitory computer-readable storage medium storing a program for causing a computer to execute a method, the method comprising:
having an image for learning for which a dividing pattern is expressed by a hierarchical structure for each predetermined area, input thereto, and generating, based on the input image, for the predetermined area, a hierarchy map in which a value indicative of a block size is associated with each of a plurality of blocks in the predetermined area; and
performing machine learning such that a mean squared error between teacher data associated with the image for learning and the hierarch map is reduced.
