# US 2021/0281849 A1 · Image processing device, image processing method, and program

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0281849 A1
- Document ID: US20210281849A1
- Status: Published application
- Kind: A1
- Application number: 17/255,159
- Filed: 2019-05-09
- Published: 2021-09-09
- Earliest claimed: 2018-07-06
- Assignees: Sony Corporation
- Applicants: Sony Corporation
- Inventors: Kenji KONDO
- Web page: https://patentgrep.com/patent/US20210281849A1

## CPC

- H04N 19/139: Analysis of motion vectors, e.g. their magnitude, direction, variance or reliability
- H04N 19/109: among a plurality of temporal predictive coding modes
- H04N 19/124: Quantisation
- H04N 19/17: the unit being an image region, e.g. an object
- H04N 19/176: the region being a block, e.g. a macroblock
- H04N 19/54: using feature points or meshes
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

When a motion detecting unit (122a) (a motion compensating unit), which has a plurality of motion compensation modes for compensating the state of the motion occurring with time in a partial area representing some part of an image, detects the state of the motion occurring in the partial area, and when the state of the motion detected by the motion detecting unit satisfies a predetermined condition; a motion compensation execution control unit (122c) (an execution control unit) makes the motion detecting unit (122a) skip the motion compensation mode corresponding to the predetermined condition. The application concerned can be applied in, for example, an image coding device.

## Claims

### Claim 1

An image processing device comprising:
a motion compensating unit that
has a plurality of motion compensation modes for compensating state of motion occurring with time in a partial area representing some part of an image,
detects state of motion occurring in the partial area, and
compensates the detected state of motion and generates a predicted image; and
an execution control unit that, either when the state of motion detected by the motion compensating unit satisfies a predetermined condition or when condition under which the motion compensating unit generates the predicted image satisfies the predetermined condition, makes the motion compensating unit skip motion compensation mode corresponding to the predetermined condition.

### Claim 2 (depends on 1)

The image processing device according to claim 1, further comprising a condition determining unit that,
based on direction and length of motion vectors at maximum of three apices of a rectangular partial area detected by the motion compensating unit, and
based on width and height of the partial area,
determines whether state of motion of the partial area satisfies the predetermined condition.

### Claim 3 (depends on 1)

The image processing device according to claim 1, wherein
the predetermined condition indicates that state of motion of the partial area involves translation and rotation, and
when the predetermined condition is satisfied, the execution control unit makes the motion compensating unit skip
a translation-scaling mode in which motion involving translation and enlargement-reduction is compensated, and
an affine transformation mode in which motion involving translation, rotation, enlargement-reduction, and skew deformation is compensated.

### Claim 4 (depends on 1)

The image processing device according to claim 1, wherein
the predetermined condition indicates that state of motion of the partial area involves translation and enlargement-reduction, and
when the predetermined condition is satisfied, the execution control unit makes the motion compensating unit skip
a translation-rotation mode in which motion involving translation and rotation is compensated, and
an affine transformation mode in which motion involving translation, rotation, enlargement-reduction, and skew deformation is compensated.

### Claim 5 (depends on 1)

The image processing device according to claim 1, wherein
the predetermined condition indicates that state of motion of the partial area involves translation, rotation, enlargement-reduction, and skew deformation, and
when the predetermined condition is satisfied, the execution control unit makes the motion compensating unit skip
a translation-scaling mode in which motion involving translation and enlargement-reduction is compensated, and
a translation-rotation mode in which motion involving translation and rotation is compensated.

### Claim 6 (depends on 1)

The image processing device according to claim 1, wherein
when the motion compensating unit uses result of motion compensation performed in a plurality of neighboring areas which are positioned in neighborhood of the partial areas and in which motion compensation is already performed, and compensates state of motion of the partial area so as to generate the predicted image,
the execution control unit detects state of motion in the partial area based on
frequency of occurrence of motion compensation modes used for motion compensation in the plurality of neighboring areas, and
costs indicating extent of prediction according to predicted images that are generated when motion compensation is performed by applying, to the partial area, the motion compensation modes used for motion compensation in the plurality of neighboring areas.

### Claim 7 (depends on 6)

The image processing device according to claim 6, wherein the motion compensating unit calculates the costs in order of frequency of occurrence of the motion compensation modes in the plurality of neighboring areas.

### Claim 8 (depends on 6)

The image processing device according to claim 6, wherein
the predetermined condition indicates that state of motion of the partial area involves translation and rotation, and
when the predetermined condition is satisfied, the execution control unit makes the motion compensating unit skip
a translation-scaling mode in which motion involving translation and enlargement-reduction is compensated, and
an affine transformation mode in which motion involving translation, rotation, enlargement-reduction, and skew deformation is compensated.

### Claim 9 (depends on 6)

The image processing device according to claim 6, wherein
the predetermined condition indicates that state of motion of the partial area involves translation and enlargement-reduction, and
when the predetermined condition is satisfied, the execution control unit makes the motion compensating unit skip
a translation-rotation mode in which motion involving translation and rotation is compensated, and
an affine transformation mode in which motion involving translation, rotation, enlargement-reduction, and skew deformation is compensated.

### Claim 10 (depends on 6)

The image processing device according to claim 6, wherein
the predetermined condition indicates that state of motion of the partial area involves translation, rotation, enlargement-reduction, and skew deformation, and
when the predetermined condition is satisfied, the execution control unit makes the motion compensating unit skip
a translation-scaling mode in which motion involving translation and enlargement-reduction is compensated, and
a translation-rotation mode in which motion involving translation and rotation is compensated.

### Claim 11 (depends on 1)

The image processing device according to claim 1, wherein
the predetermined condition indicates that size of the predetermined area is smaller than a predetermined size, and
when the predetermined condition is satisfied, the execution control unit makes the motion compensating unit skip predetermined motion compensation.

### Claim 12 (depends on 1)

The image processing device according to claim 1, wherein
the predetermined condition indicates that size of the predetermined area is smaller than a predetermined size, and
when the predetermined condition is satisfied, the execution control unit makes the motion compensating unit skip motion compensation modes other than
a translation mode in which motion involving translation is compensated,
a translation-rotation mode in which motion involving translation and rotation is compensated, and
a translation-scaling mode in which motion involving translation and enlargement-reduction is compensated.

### Claim 13 (depends on 1)

The image processing device according to claim 1, wherein
the predetermined condition indicates that size of the predetermined area is equal to or greater than a predetermined size, and
when the predetermined condition is satisfied, the execution control unit makes the motion compensating unit skip motion compensation modes other than a motion compensation mode which has lowest cost from among costs that represent extent of prediction according to predicted images generated as a result of performing motion compensation in the partial area by applying a plurality of motion compensation modes provided in the motion compensating unit.

### Claim 14 (depends on 1)

The image processing device according to claim 1, wherein
the predetermined condition indicates that a quantization parameter, which is used in quantizing result of motion compensation, is smaller than a predetermined value, and
when the predetermined condition is satisfied, the execution control unit makes the motion compensating unit skip
a translation-scaling mode in which motion involving translation and enlargement-reduction is compensated, and
a translation-rotation mode in which motion involving translation and rotation is compensated.

### Claim 15 (depends on 1)

The image processing device according to claim 1, wherein
the predetermined condition indicates that
a quantization parameter, which is used in quantizing result of motion compensation, is smaller than a predetermined value, and
a cost which represents extent of prediction according to a predicted image generated as a result of performing motion compensation in the partial area by applying an affine transformation mode, in which motion involving translation, rotation, enlargement-reduction, and skew deformation is compensated, is smaller than a predetermined threshold value, and
when the predetermined condition is satisfied, the execution control unit makes the motion compensating unit skip motion compensation modes other than the affine transformation mode in which motion involving translation, rotation, enlargement-reduction, and skew deformation is compensated.

### Claim 16 (depends on 1)

The image processing device according to claim 1, wherein
the predetermined condition indicates that
a quantization parameter, which is used in quantizing result of motion compensation, is smaller than a predetermined value, and
a cost which represents extent of prediction according to a predicted image generated as a result of performing motion compensation in the partial area by applying an affine transformation mode, in which motion involving translation, rotation, enlargement-reduction, and skew deformation is compensated, is equal to or greater than a predetermined threshold value, and
when the predetermined condition is satisfied, the execution control unit makes the motion compensating unit skip motion compensation modes other than a translation mode in which motion involving translation is compensated.

### Claim 17 (depends on 1)

The image processing device according to claim 1, wherein
the predetermined condition indicates that a quantization parameter, which is used in quantizing result of motion compensation, is equal to or greater than a predetermined value, and
when the predetermined condition is satisfied, the execution control unit makes the motion compensating unit skip motion compensation modes other than a motion compensation mode which has lowest cost from among costs that represent extent of prediction according to predicted images generated as a result of performing motion compensation in the partial area by applying a plurality of motion compensation modes.

### Claim 18

An image processing method in which
a plurality of motion compensation modes is provided for compensating state of motion occurring with time in a partial area representing some part of an image,
state of motion occurring in the partial area is detected, and
the detected state of motion is compensated and a predicted image is generated,
the image processing method comprising:
skipping that, either when state of motion detected in the partial area satisfies a predetermined condition or when condition for generating the predicted image satisfies the predetermined condition, includes skipping motion compensation mode corresponding to the predetermined condition.

### Claim 19

A program that causes a computer, which is included in an image processing device, to function as:
a motion compensating unit that
has a plurality of motion compensation modes for compensating state of motion occurring with time in a partial area representing some part of an image,
detects state of motion occurring in the partial area, and
compensates the detected state of motion and generates a predicted image; and
an execution control unit that, either when the state of motion detected by the motion compensating unit satisfies a predetermined condition or when condition under which the motion compensating unit generates the predicted image satisfies the predetermined condition, makes the motion compensating unit skip motion compensation mode corresponding to the predetermined condition.
