# US 2008/0278633 A1 · Image processing method and image processing apparatus

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2008/0278633 A1
- Document ID: US20080278633A1
- Status: Published application
- Kind: A1
- Application number: 11/798,035
- Filed: 2007-05-09
- Published: 2008-11-13
- Earliest claimed: 2007-05-09
- Inventors: Mikhail Tsoupko-Sitnikov; Igor Borovikov; Shinichi Yamashita; Masuharu Endo
- Web page: https://patentgrep.com/patent/US20080278633A1

## CPC

- H04N 5/144: {Movement detection (for video coding H04N19/503; analysis of motion in general G06T7/20)}
- G06T 7/238: using non-full search, e.g. three-step search
- G06T 2207/10016: Video; Image sequence
- G06T 2207/20016: Hierarchical, coarse-to-fine, multiscale or multiresolution image processing; Pyramid transform
- H04N 19/119: Adaptive subdivision aspects, e.g. subdivision of a picture into rectangular or non-rectangular coding blocks
- H04N 19/139: Analysis of motion vectors, e.g. their magnitude, direction, variance or reliability
- H04N 19/176: the region being a block, e.g. a macroblock
- H04N 19/553: Motion estimation dealing with occlusions
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

A corresponding point information generator computes matching between a source image frame and a destination image frame in image data comprising consecutive image frames so as to determine corresponding point information indicating pixel-by-pixel matching. A motion vector detector determines a motion vector for each pixel in the source image frame according to a result of matching. A reliability area isolating unit segments an image frame in which a motion vector is determined into blocks, so as to isolate, in each block, a reliable area characterized by relatively high precision of the motion vector as calculated and a non-reliable area characterized by relatively low precision of the motion vector. A motion vector improving unit calculates, when a motion vector of a reliable area is applied to a pixel in a non-reliable area adjacent to the reliable area, an error between a pixel value occurring at the destination as a result of application and a pixel value of a corresponding pixel in the destination image frame, and, when the error is equal to or smaller than a threshold, incorporates the pixel in the non-reliable area into the reliable area, and replaces the motion vector of that pixel by the motion vector of the reliable area.

## Claims

### Claim 1

An image processing method comprising:
computing matching between two image frames in image data comprising consecutive image frames so as to determine corresponding point information indicating pixel-by-pixel correspondence; and
for a pixel characterized by relatively low reliability of correspondence, performing block matching between images so as to determine correspondence block by block.

### Claim 2

An image processing method comprising:
initial matching in which correspondence point information is determined for each pixel in a source image frame and in a destination image frame in image data comprising consecutive image frames;
determining a motion vector according to a result of matching and determining for each pixel the reliability of the motion vector thus determined; and
for a pixel characterized by relatively low reliability of the motion vector as determined, performing block matching between blocks in the source image frame and in the destination image frame, and determining an updated motion vector by re-calculation, each block comprising a plurality of pixels.

### Claim 3

An image processing apparatus comprising:
a matching processor which computes matching between a source image frame and a destination image frame in image data comprising consecutive image frames so as to determine corresponding point information indicating pixel-by-pixel correspondence;
a motion vector detector which determines a motion vector for each pixel in the source image frame, according to a result of matching;
a reliability area isolating unit which segments an image frame in which a motion vector is determined into blocks, so as to partition into a reliable area characterized by relatively high precision of the motion vector as calculated and a non-reliable area characterized by relatively low precision of the motion vector; and
a motion vector improving unit which calculates, when a motion vector of a reliable area is applied to a pixel in a non-reliable area adjacent to the reliable area, an error between a pixel value occurring at the destination as a result of application and a pixel value of a corresponding pixel in the destination image frame, and, when the error is equal to or smaller than a threshold, incorporates the pixel in the non-reliable area into the reliable area, and replaces the motion vector of that pixel by the motion vector of the reliable area.

### Claim 4 (depends on 3)

The image processing apparatus according to claim 3, further comprising:
a block matching unit which performs block matching on blocks with pixels, which are included in the non-reliable area of the source image frame and which are not incorporated into the reliable area by the motion vector improving unit, so as to exhaustively search for a block in the destination image frame characterized by the smallest matching error, wherein
the motion vector of the block subjected to matching is replaced by the motion vector identified as a result of block matching.

### Claim 5

An image processing apparatus comprising:
a motion vector detector which determines motion vectors in a forward direction and in a reverse direction between a source image frame and a destination image frame in image data comprising consecutive image frames; and
an occlusion detector which compares the motion vector in the forward direction with the motion vector in the reverse direction, and, when there is a pixel characterized by a difference between the two, determines that the pixel is included in either i) an area in which an object including the pixel inside is hidden by another object within the same frame, or ii) an occlusion area in which the object including the pixel inside hides another object within the same frame behind.

### Claim 6 (depends on 5)

The image processing apparatus according to claim 5, wherein
the occlusion detector processes a pixel for which the size of the motion vector is zero either in the forward direction or the reverse direction such that the detector determines an area including the pixel to be a covered area hidden by an occluder, when the source image frame includes a point corresponding to the pixel but the destination image frame does not include a point corresponding to the pixel, and the detector determines the area including the pixel to be an uncovered area that presents itself from behind an occluder, when the source image frame does not include a point corresponding to the pixel but the destination image frame includes a point corresponding to the pixel.

### Claim 7 (depends on 6)

The image processing apparatus according to claim 6, wherein
a sum of sets of the covered area and the uncovered area is used as a mask to be applied to an image frame.

### Claim 8 (depends on 7)

The image processing apparatus according to claim 7, further comprising:
an edge detector which detects an edge between an object and a background in an image frame, so as to create an edge image; and
and edge extractor which removes an edge between stationary objects in the source image frame or the destination image frame by calculating a sum of sets of the mask and the edge image, so as to extract only an edge portion between a moving object and a stationary object.

### Claim 9

A computer program product comprising:
initial matching in which correspondence point information is determined for each pixel in a source image frame and in a destination image frame in image data comprising consecutive image frames;
determining a motion vector according to a result of matching and determining for each pixel the reliability of the motion vector thus determined; and
for a pixel characterized by relatively low reliability of the motion vector thus determined, performing block matching between blocks in the source image frame and in the destination image frame, and determining an updated motion vector by re-calculation, each block comprising a plurality of pixels.

### Claim 10

An image processing method comprising:
computing matching between two image frames in image data comprising consecutive image frames so as to determine corresponding point information indicating correspondence between the image frames; and
determining a motion vector for each pixel according to a result of matching;
detecting, on the basis of the motion vector thus calculated, an area in which an object is hidden in a frame by another frame within the same frame and an occlusion area in which an object hides another object within the same frame; and
isolating between a stationary portion and a moving portion in an image frame, on the basis of the motion vector and the occlusion area.
