# US 12,010,456 B2 · Method for performing frame interpolation based on single-directional motion and associated non-transitory machine-readable medium > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,010,456 B2 - Document ID: US12010456B2 - Status: Granted - Kind: B2 - Application number: 18/128,180 - Filed: 2023-03-29 - Published: 2023-10-12 - Granted: 2024-06-11 - Earliest claimed: 2022-04-06 - Assignees: Mediatek Inc. - Applicants: Mediatek Inc. - Inventors: Yao-Sheng Wang; Pei-Kuei Tsung; Chia-Ni Lu; Yu-Sheng Lin; Chien-Yu Huang; Chih-Wen Goo; Cheng-Lung Jen - Web page: https://patentgrep.com/patent/US12010456B2 ## CPC - G06T 3/4007: based on interpolation, e.g. bilinear interpolation (image demosaicing G06T3/4015; edge-driven or edge-based scaling G06T3/403) - H04N 7/014: {involving the use of motion vectors (motion estimation and compensation in video coding H04N19/51)} - H04N 7/0127: {by changing the field or frame frequency of the incoming video signal, e.g. frame rate converter} - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract A frame interpolation method for generating a third image frame interpolated between a first image frame and a second image frame includes: performing motion estimation upon a first input image frame and a second input image frame, to obtain a single-directional motion, wherein the first input image frame is derived from the first image frame, and the second input image frame is derived from the second image frame; scaling the single-directional motion according to a time point of the third image frame, to generate a scaled motion; deriving a forward-warped result from a result of performing a forward warping operation and a first inverse operation upon the scaled motion; performing a second inverse operation upon the forward-warped result, to generate an inversed result; and generating the third image frame according to the first image frame, the second image frame, the forward-warped result, and the inversed result. ## Claims ### Claim 1 A frame interpolation method for generating a third image frame interpolated between a first image frame and a second image frame, comprising: performing motion estimation upon a first input image frame and a second input image frame, to obtain a single-directional motion, wherein the first input image frame is derived from the first image frame, and the second input image frame is derived from the second image frame; scaling the single-directional motion according to a time point of the third image frame, to generate a scaled motion; deriving a forward-warped result from a result of performing a forward warping operation and a first inverse operation upon the scaled motion; performing a second inverse operation upon the forward-warped result, to generate an inversed result; and generating the third image frame according to the first image frame, the second image frame, the forward-warped result, and the inversed result. ### Claim 2 (depends on 1) The frame interpolation method of claim 1, further comprising: resizing the first image frame and the second image frame, respectively, to generate a first resized image frame as the first input image frame and a second resized image frame as the second input image frame. ### Claim 3 (depends on 1) The frame interpolation method of claim 1, wherein the step of deriving the forward-warped result from the result of performing the forward warping operation and the first inverse operation upon the scaled motion comprises: performing the forward warping operation and the first inverse operation upon the scaled motion, to generate a coarse forward-warped result; and refining the coarse forward-warped result according to at least the first input frame and the second input frame, to generate the forward-warped result. ### Claim 4 (depends on 3) The frame interpolation method of claim 3, further comprising: performing a backward warping operation upon the first input image frame according to the coarse forward-warped result, to generate a backward-warped result; wherein the step of refining the coarse forward-warped result according to at least the first input image frame and the second input image frame, to generate the forward-warped result comprises: refining the coarse forward-warped result according to the first input image frame, the second input image frame, and the backward-warped result, to generate the forward-warped result. ### Claim 5 (depends on 3) The frame interpolation method of claim 3, wherein the coarse forward-warped result comprises an alpha map and a forward-warped motion, and the step of refining the coarse forward-warped result according to at least the first input image frame and the second input image frame, to generate the forward-warped result comprises: refining the alpha map according to at least the first input image frame and the second input image frame, to generate a refined alpha map; and refining the forward-warped motion according to at least the first input image frame and the second input image frame, to generate a refined motion; wherein the forward-warped result comprises the refined alpha map and the refined motion. ### Claim 6 (depends on 5) The frame interpolation method of claim 5, wherein the step of performing the second inverse operation upon the forward-warped result, to generate the inversed result comprises: performing the second inverse operation upon the refined alpha map, to generate an inversed alpha map; and performing the second inverse operation upon the refined motion, to generate an inversed motion; wherein the inversed result comprises the inversed alpha map and the inversed motion. ### Claim 7 (depends on 3) The frame interpolation method of claim 3, wherein the step of refining the coarse forward-warped result according to at least the first input image frame and the second input image frame, to generate the forward-warped result further comprises: inpainting holes caused by performing the forward warping operation. ### Claim 8 (depends on 1) The frame interpolation method of claim 1, wherein the step of generating the third image frame according to the first image frame, the second image frame, the forward-warped result, and the inversed result comprises: performing a backward warping operation upon the first image frame and the second frame according to the forward-warped result and the inversed result, respectively, to generate a first backward-warped result and a second backward-warped result; and blending the first backward-warped result and the second backward-warped result according to the forward-warped result and the inversed result, to generate the third image frame. ### Claim 9 (depends on 8) The frame interpolation method of claim 8, wherein the forward-warped result comprises a motion, the inversed result comprises an inversed motion, and the step of performing the backward warping operation upon the first image frame and the second image frame according to the forward-warped result and the inversed result, respectively, to generate the first backward-warped result and the second backward-warped result comprises: performing the backward warping operation upon the first image frame according to the motion, to generate the first backward-warped result; and performing the backward warping operation upon the second image frame according to the inversed motion, to generate the second backward-warped result. ### Claim 10 (depends on 8) The frame interpolation method of claim 8, wherein the forward-warped result comprises an alpha map, the inversed result comprises an inversed alpha map, and the step of blending the first backward-warped result and the second backward-warped result according to the forward-warped result and the inversed result, to generate the third image frame comprises: multiplying the first backward-warped result and the alpha map, to generate a first multiplication result; multiplying the second backward-warped result and the inversed alpha map, to generate a second multiplication result; and adding the first multiplication result and the second multiplication result to generate the third image frame. ### Claim 11 A non-transitory machine-readable medium for storing a program code, wherein when loaded and executed by a processor, the program code instructs the processor to execute: a frame interpolation module, arranged to generate a third image frame interpolated between a first image frame and a second image frame, and comprising: a motion estimating module, arranged to perform motion estimation upon a first input image frame and a second input image frame, to obtain a single-directional motion, wherein the first input image frame is derived from the first image frame, and the second input image frame is derived from the second image frame; a scaling module, arranged to scale the single-directional motion according to a time point of the third image frame, to generate a scaled motion; a first processing module, arranged to derive a forward-warped result from a result of performing a forward warping operation and a first inverse operation upon the scaled motion; an inverse module, arranged to perform a second inverse operation upon the forward-warped result, to generate an inversed result; and a second processing module, arranged to generate the third image frame according to the first image frame, the second image frame, the forward-warped result, and the inversed result. ### Claim 12 (depends on 11) The non-transitory machine-readable medium of claim 11, wherein the frame interpolation module further comprises: a resizing module, arranged to resize the first image frame and the second image frame, respectively, to generate a first resized image frame as the first input image frame and a second resized image frame as the second input image frame. ### Claim 13 (depends on 11) The non-transitory machine-readable medium of claim 11, wherein the first processing module comprises: a forward warping module, arranged to perform the forward warping operation and the first inverse operation upon the scaled motion, to generate a coarse forward-warped result; and a refining module, arranged to refine the coarse forward-warped result according to at least the first input frame and the second input frame, to generate the forward-warped result. ### Claim 14 (depends on 13) The non-transitory machine-readable medium of claim 13, wherein the frame interpolation module further comprises: a backward warping module, arranged to perform a backward warping operation upon the first input image frame according to the coarse forward-warped result, to generate a backward-warped result; wherein the refining module is further arranged to refine the coarse forward-warped result according to the first input image frame, the second input image frame, and the backward-warped result, to generate the forward-warped result. ### Claim 15 (depends on 13) The non-transitory machine-readable medium of claim 13, wherein the coarse forward-warped result comprises an alpha map and a forward-warped motion, and the refining module is further arranged to: refine the alpha map according to at least the first input image frame and the second input image frame, to generate a refined alpha map; and refine the forward-warped motion according to at least the first input image frame and the second input image frame, to generate a refined motion; wherein the forward-warped result comprises the refined alpha map and the refined motion. ### Claim 16 (depends on 15) The non-transitory machine-readable medium of claim 15, wherein the refining module is further arranged to: perform the second inverse operation upon the refined alpha map, to generate an inversed alpha map; and perform the second inverse operation upon the refined motion, to generate an inversed motion; wherein the inversed result comprises the inversed alpha map and the inversed motion. ### Claim 17 (depends on 13) The non-transitory machine-readable medium of claim 13, wherein the inverse module is further arranged to inpaint holes caused by performing the forward warping operation. ### Claim 18 (depends on 11) The non-transitory machine-readable medium of claim 11, wherein the second processing module comprises: multiple backward warping modules, arranged to perform a backward warping operation upon the first image frame and the second frame according to the forward-warped result and the inversed result, respectively, to generate a first backward-warped result and a second backward-warped result; and a blending module, arranged to blend the first backward-warped result and the second backward-warped result according to the forward-warped result and the inversed result, to generate the third image frame. ### Claim 19 (depends on 18) The non-transitory machine-readable medium of claim 18, wherein the forward-warped result comprises a motion, the inversed result comprises an inversed motion, and the multiple backward warping modules comprises: a first backward warping module, arranged to perform the backward warping operation upon the first image frame according to the motion, to generate the first backward-warped result; and a second backward warping module, arranged to perform the backward warping operation upon the second image frame according to the inversed motion, to generate the second backward-warped result. ### Claim 20 (depends on 18) The non-transitory machine-readable medium of claim 18, wherein the forward-warped result comprises an alpha map, the inversed result comprises an inversed alpha map, and the blending module is further arranged to: multiply the first backward-warped result and the alpha map, to generate a first multiplication result; multiply the second backward-warped result and the inversed alpha map, to generate a second multiplication result; and add the first multiplication result and the second multiplication result to generate the third image frame.