# US 2008/0187050 A1 · Frame interpolation apparatus and method for motion estimation through separation into static object and moving object

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2008/0187050 A1
- Document ID: US20080187050A1
- Status: Published application
- Kind: A1
- Application number: 12/020,741
- Filed: 2008-01-28
- Published: 2008-08-07
- Earliest claimed: 2007-02-02
- Assignees: Samsung Electronics Co., Ltd.
- Inventors: Hwa-hyun Cho; Jong-seon Kim
- Web page: https://patentgrep.com/patent/US20080187050A1

## CPC

- H04N 5/144: {Movement detection (for video coding H04N19/503; analysis of motion in general G06T7/20)}
- G06T 3/4007: based on interpolation, e.g. bilinear interpolation (image demosaicing G06T3/4015; edge-driven or edge-based scaling G06T3/403)
- H04N 7/014: {involving the use of motion vectors (motion estimation and compensation in video coding H04N19/51)}
- H04N 7/0127: {by changing the field or frame frequency of the incoming video signal, e.g. frame rate converter}
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

Provided are frame interpolation apparatus and methods in which motion estimation is performed by separation into a static object and a moving object. The frame interpolation apparatus interpolates multiple frames including an nth frame and an (n−1)th frame located adjacent the nth frame in order to generate an interpolation frame. Some embodiments of the apparatus include a static object separation unit, a motion vector (MV) estimation unit, and an interpolation frame generation unit. The static object separation unit may compare a macroblock (MB) of the nth frame with an MB of the (n−1)th frame, which may correspond to the MB of the nth frame, in order to separate each MB of the nth frame into a static object and a moving object. The MV estimation unit may search in the (n−1)th frame for an MB that matches with each of MBs of the nth frame, which may be determined to be the moving object, to estimate an MV. The interpolation frame generation unit may generate the interpolation frame using the estimated MV, the nth frame, and the (n−1)th frame.

## Claims

### Claim 1

A frame interpolation apparatus that interpolates a plurality of frames including an nth frame and an (n−1)th frame located adjacent to the nth frame to generate an interpolation frame, the frame interpolation apparatus comprising:
a static object separation unit that is configured to compare an nth frame macroblock (MB) of a plurality of no frame MBs with an (n−1)th frame MB of a plurality of (n−1)th frame MBs to separate the nth frame MB into a static object and a moving object;
a motion vector (MV) estimation unit that is configured to search the (n−1)th frame for an MB that matches the nth frame MB that is determined to be the moving object to estimate an MV; and
an interpolation frame generation unit that is configured to generate the interpolation frame using the estimated MV, the nth frame, and the (n−1)th frame.

### Claim 2 (depends on 1)

The frame interpolation apparatus of claim 1, wherein the MV estimation unit comprises:
a search range processing unit that is configured to search in a first search range of the (n−1)th frame for the MB that matches with the nth frame MB to estimate a first MV;
a modified search range processing unit that is configured to search in a second search range of the (n−1)th frame for the MB that matches with the nth frame MB to estimate a second MV; and
a current MB processing unit that is configured to match nth frame MB with an (n−1)th frame MB of the plurality of (n−1)th frame MBs, which corresponds to the nth frame MB to estimate a third MV,
wherein the first search range includes MBs of the (n−1)th frame and the second search range includes the (n−1)th frame MBs that correspond to nth frame MBs that are determined to be the moving object.

### Claim 3 (depends on 2)

The frame interpolation apparatus of claim 2, wherein the first search range includes the (n−1)th frame MB that corresponds to the matching nth frame MB and the (n−1)th frame MBs that are adjacent the corresponding (n−1)th frame MB.

### Claim 4 (depends on 2)

The frame interpolation apparatus of claim 2, wherein the second search range includes the (n−1)th frame MB that corresponds to the matching nth frame MB and (n−1)th frame MBs that adjacent the corresponding (n−1)th frame MB and that corresponds to the nth frame MBs determined to be moving objects.

### Claim 5 (depends on 2)

The frame interpolation apparatus of claim 2, wherein the search range processing unit is configured to calculate a sum of absolute differences (SAD) for matching MBs and to estimate the first MV between an (n−1)th frame MB that has a minimum SAD and the nth frame MB.

### Claim 6 (depends on 2)

The frame interpolation apparatus of claim 2, wherein the modified search range processing unit is configured to calculate a sum of absolute differences (SAD) for matching MBs and to estimate the second MV between an (n−1)th frame MB that has a minimum SAD, and the nth frame MB.

### Claim 7 (depends on 2)

The frame interpolation apparatus of claim 2, wherein the current MB processing unit is configured to calculate a sum of absolute differences (SAD) for matching MBs to estimate the third MV.

### Claim 8 (depends on 2)

The frame interpolation apparatus of claim 2, wherein the interpolation frame generation unit is configured to generate the interpolation frame using a median value of the estimated first MV, the estimated second MV, and the estimated third MV.

### Claim 9 (depends on 1)

The frame interpolation apparatus of claim 1, wherein the interpolation frame generation unit is configured to generate the interpolation frame using a median value of the nth frame MB and the (n−1)th frame MB when a matching MB that matches with the nth frame MB of exists in the (n−1)th frame.

### Claim 10 (depends on 1)

The frame interpolation apparatus of claim 1, wherein the interpolation frame generation unit is configured to generate the interpolation frame using the nth frame MB or the (n−1)th frame MB according to a direction of the estimated MV when a matching MB that matches with the nth frame MB does not exist in the (n−1)th frame.

### Claim 11 (depends on 10)

The frame interpolation apparatus of claim 10, wherein the interpolation frame generation unit uses the nth frame MB when a direction of the estimated MV is from left to right and uses the (n−1)th frame MB when a direction of the estimated MV is from right to left.

### Claim 12 (depends on 1)

The frame interpolation apparatus of claim 1, wherein the static object separation unit is configured to calculate a difference between luminance signals of the nth frame MB and a corresponding (n−1)th frame MB, compare the difference with a threshold that is a criterion for distinguishing the moving object from the static object, and separate each of the plurality of nth frame MBs into the static object and the moving object.

### Claim 13 (depends on 12)

The frame interpolation apparatus of claim 12, wherein the static object separation unit is configured to determine the nth frame MB as the moving object when the difference is greater than the threshold and to determine the nth frame MB as the static object when the difference is less than the threshold.

### Claim 14 (depends on 1)

The frame interpolation apparatus of claim 1, wherein the MV estimation unit is configured to search in a search range of the (n−1)th frame for an MB that matches with an MB of the nth frame to estimate the MV, wherein the search range includes an (n−1)th frame MB that corresponds to the nth frame MB and the (n−1)th frame MBs that are adjacent the (n−1)th frame MB that corresponds to the nth frame MB.

### Claim 15

A frame interpolation apparatus that interpolates a plurality of frames including an nth frame and an (n−1)th frame located adjacent the nth frame in order to generate an interpolation frame, the frame interpolation apparatus comprising:
a static object separation unit that is configured to compare an nth frame macroblock (MB) with an (n−1)th frame MB that corresponds to the nth frame MB to separate each of a plurality of nth frame MBs into a static object and a moving object;
a motion vector (MV) estimation unit that is configured to search in a search range of the (n−1)th frame for an MB that matches each of the plurality of nth frame MBs that are determined to be the moving object to estimate an MV; and
an interpolation frame generation unit that is configured to generate the interpolation frame using the estimated MV, the nth frame, and the (n−1)th frame,
wherein the search range includes the (n−1)th frame CABs that correspond to each of the plurality of nth frame MBs that are determined to be the moving object.

### Claim 16 (depends on 15)

The frame interpolation apparatus of claim 15, wherein the search range includes the (n−1)th frame MB that corresponds to a matching nth frame MB and the (n−1)th frame MB's that are located adjacent a corresponding (n−1)th frame MB and that correspond to the nth frame MBs that are determined to be the moving object.

### Claim 17 (depends on 15)

The frame interpolation apparatus of claim 15, wherein the interpolation frame generation unit is configured to generate the interpolation frame using a median value of the nth frame MB and the (n−1)th frame MB when an MB that matches the nth frame MB exists in the (n−1)th frame.

### Claim 18 (depends on 15)

The frame interpolation apparatus of claim 15, wherein the interpolation frame generation unit is configured to generate the interpolation frame using the nth frame MB or the (n−1)th frame MB according to a direction of the estimated MV when an (n−1)th frame MB that matches with the nth frame MB does not exist in the (n−1)th frame.

### Claim 19 (depends on 15)

The frame interpolation apparatus of claim 15, wherein the static object separation unit is configured to calculate a difference between luminance signals of the nth frame MB and a corresponding (n−1)th frame MB, to compare the difference with a threshold that is a criterion for distinguishing the moving object from the static object, and to separate each nth frame MB into the static object and the moving object.

### Claim 20

A frame interpolation method that interpolates a plurality of frames including an nth frame and an (n−1)th frame that is located adjacent the nth frame to generate an interpolation frame, the frame interpolation method comprising:
comparing an nth frame macroblock (MB) with an (n−1)th frame MB that corresponds to the nth frame MB to separate each of a plurality of nth frame MBs into a static object and a moving object;
searching in the (n−1)th frame for an MB that matches each of the plurality of nth frame MBs that are determined to be the moving object;
estimating a motion vector (MV) responsive to the MB that matches each of the plurality of nth frame MBs that are determined to be the moving object; and
generating the interpolation frame using an estimated MV, the nth frame, and the (n−1)th frame.

### Claim 21 (depends on 20)

The frame interpolation method of claim 20, wherein estimating the MV comprises:
searching in a first search range of the (n−1)th frame for an MB that matches each of the plurality of nth frame MBs to estimate a first MV;
searching in a second search range of the (n−1)th frame for an MB that matches each of the plurality of the nth frame MBs to estimate a second MV; and
matching an nth frame MB with an (n−1)th frame MB that corresponds to the nth frame MB to estimate a third MV,
wherein the first search range includes the (n−1)th frame MBs and the second search range includes the (n−1)th frame MBs that correspond to the nth frame MBs that are determined to be the moving object.

### Claim 22 (depends on 21)

The frame interpolation method of claim 21, wherein the first search range includes the (n−1)th frame MB that corresponds to the matching nth frame MB and the plurality of the (n−1)th frame MBs that are located adjacent to the corresponding (n−1)th frame MB.

### Claim 23 (depends on 21)

The frame interpolation method of claim 21, wherein the second search range includes the (n−1)th frame MB that corresponds to a matching nth frame MB and the plurality of (n−1)th frame MBs that are adjacent the corresponding (n−1)th frame MB and correspond to the plurality of nth frame MBs that are determined to be the moving object.

### Claim 24 (depends on 21)

The frame interpolation method of claim 21, wherein estimation of the first MV comprises:
calculating a sum of absolute differences (SAD) for matching MBs; and
estimating the first MV between an (n−1)th frame MB that has a minimum SAD and the nth frame MB.

### Claim 25 (depends on 21)

The frame interpolation method of claim 21, wherein estimation of the second MV comprises:
calculating a sum of absolute differences (SAD) for matching MBs; and
estimating the second MV between an (n−1)th frame MB that has the minimum SAD and the nth frame MB.

### Claim 26 (depends on 21)

The frame interpolation method of claim 21, wherein estimation of the third MV comprises calculating a sum of absolute differences (SAD) for matching MBs to estimate the third MV.

### Claim 27 (depends on 21)

The frame interpolation method of claim 21, wherein generating the interpolation frame comprises:
selecting a median value of the estimated first MV, the estimated second MV, and the estimated third MV; and
generating the interpolation frame using the selected median value.

### Claim 28 (depends on 20)

The frame interpolation method of claim 20, wherein generating the interpolation frame comprises:
determining whether an MB that matches the nth frame MB exists in the (n−1)th frame;
generating the interpolation frame using a median value of the nth frame MB and the (n−1)th frame MB when an MB that matches the nth frame MB exists in the (n−1)th frame; and
generating the interpolation frame using the nth frame MB or the (n−1)th frame MB according to a direction of the estimated MV when an MB that matches the nth frame MB does not exist in the (n−1)th frame.

### Claim 29 (depends on 28)

The frame interpolation method of claim 28, wherein generating the interpolation frame according to the direction of the estimated MV comprises:
determining the direction of the estimated MV;
generating the interpolation frame using the nth frame MB when the direction of the estimated MV is from left to right; and
generating the interpolation frame using the (n−1)th frame MB when the direction of the estimated MV is from right to left.

### Claim 30 (depends on 20)

The frame interpolation method of claim 20, wherein comparing an nth frame macroblock (MB) with an (n−1)th frame MB that corresponds to the nth frame MB to separate each of a plurality of nth frame MBs into a static object and a moving object comprises:
calculating a difference between luminance signals of the nth frame MB and the corresponding (n−1)th frame MB;
comparing the difference with a threshold that is a criterion for distinguishing the moving object from the static object; and
separating each of the plurality of nth frame MBs into the static object and the moving object.

### Claim 31 (depends on 30)

The frame interpolation method of claim 30, wherein separating each of the plurality of nth frame MBs comprises:
determining the nth frame MB as the moving object when the difference is greater than the threshold; and
determining the nth frame MB as the static object when the difference is less than the threshold.

### Claim 32 (depends on 20)

The frame interpolation method of claim 20, wherein estimation of the MV comprises searching in a search range of the (n−1)th frame for an MB that matches an nth frame MB to estimate the MV, wherein the search range includes an (n−1)th frame MB that corresponds to a matching nth frame MB and (n−1)th frame MBs that are adjacent a corresponding (n−1)th frame MB.

### Claim 33

A frame interpolation method that interpolates a plurality of frames including an nth frame and an (n−1)th frame that is located adjacent the nth frame to generate an interpolation frame, the frame interpolation method comprising:
comparing an nth frame macroblock (MB) in the nth frame with an (n−1)th frame MB in the (n−1)th frame that corresponds to the nth frame MB to separate each of a plurality of nth frame MBs into a static object and a moving object;
searching in a search range of the (n−1)th frame for an MB that matches with each of the plurality of nth frame MBs that are determined to be the moving object to estimate a motion vector (MV); and
generating the interpolation frame using an estimated MV, the nth frame, and the (n−1)th frame,
wherein the search range includes the (n−1)th frame MBs that correspond to the nth frame MBs that are determined to be the moving object.

### Claim 34 (depends on 33)

The frame interpolation method of claim 33, wherein the search range includes the (n−1)th frame MB that corresponds to a matching nth frame MB and a plurality of (n−1)th frame MBs that are adjacent a corresponding (n−1)th frame MB and that correspond to the nth frame MBs that are determined to be the moving object.

### Claim 35 (depends on 33)

The frame interpolation method of claim 33, wherein generating the interpolation frame comprises:
determining whether an MB that matches the nth frame MB exists in the (n−1)th frame;
generating the interpolation frame using a median value of the nth frame MB and the (n−1)th frame MB when the MB that matches the nth frame MB exists in the (n−1)th frame; and
generating the interpolation frame using the nth frame MB or the (n−1)th frame MB according to a direction of the estimated MV when the MB that matches the nth frame MB does not exist in the (n−1)th frame.

### Claim 36 (depends on 33)

The frame interpolation method of claim 33, wherein generating the interpolation frame according to a direction of the estimated MV comprises:
determining the direction of the estimated MV;
generating the interpolation frame using the nth frame MB when the direction of the estimated MV is from left to right; and
generating the interpolation frame using the (n−1)th frame MB when the direction of the estimated MV is from night to left.

### Claim 37 (depends on 33)

The frame interpolation method of claim 33, wherein comparing an nth frame macroblock (MB) in the nth frame with an (n−1)th frame MB in the (n−1)th frame that corresponds to the nth frame MB to separate each of a plurality of nth frame MBs into a static object and a moving object comprises:
calculating a difference between luminance signals of the nth frame MB and a corresponding (n−1)th frame MB;
comparing the difference with a threshold that is a criterion for distinguishing the moving object from the static object; and
separating each of the plurality of nth frame MBs into the static object and the moving object.
