# US 2021/0279840 A1 · Systems and methods for multi-frame video frame interpolation > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2021/0279840 A1 - Document ID: US20210279840A1 - Status: Published application - Kind: A1 - Application number: 17/102,114 - Filed: 2020-11-23 - Published: 2021-09-09 - Earliest claimed: 2020-03-05 - Assignees: Huawei Technologies Co., Ltd. - Inventors: Zhixiang CHI; Rasoul MOHAMMADI NASIRI; Zheng LIU; Jin TANG; Juwei LU - Web page: https://patentgrep.com/patent/US20210279840A1 ## CPC - G06T 3/40: Scaling of whole images or parts thereof, e.g. expanding or contracting - G06T 7/251: {involving models} - G06N 3/045: Combinations of networks - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/0475: Generative networks - G06N 3/08: Learning methods - G06N 3/084: Backpropagation, e.g. using gradient descent - G06N 3/09: Supervised learning - G06N 3/094: Adversarial learning - G06T 7/269: using gradient-based methods - G06N 3/048: Activation functions - G06T 2207/10016: Video; Image sequence - G06T 2207/20081: Training; Learning - G06T 2207/20084: Artificial neural networks [ANN] - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS ## Abstract Systems and methods for multi-frame video frame interpolation. Higher-order motion modeling, such as cubic motion modeling, achieves predictions of intermediate optical flow between multiple interpolated frames, assisted by relaxation of the constraints imposed by the loss function used in initial optical flow estimation. A temporal pyramidal optical flow refinement module performs coarse-to-fine refinement of the optical flow maps used to generate the intermediate frames, focusing a proportionally greater amount of refinement attention to the optical flow maps for the high-error middle frames. A temporal pyramidal pixel refinement module performs coarse-to-fine refinement of the generated intermediate frames, focusing a proportionally greater amount of refinement attention to the high-error middle frames. A generative adversarial network (GAN) module calculates a loss function for training the neural networks used in the optical flow estimation module, temporal pyramidal optical flow refinement module, and/or temporal pyramidal pixel refinement module. ## Claims ### Claim 1 A method for multi-frame video frame interpolation, the method comprising: obtaining a plurality of optical flow maps corresponding to a plurality of times between a beginning time and an ending time; processing the plurality of optical flow maps to generate a first refined optical flow map corresponding to a first intermediate time proximal to the beginning time or ending time; processing the plurality of optical flow maps and the first refined intermediate optical flow map to generate a second refined optical flow map corresponding to a second intermediate time distal from both the beginning time and ending time; generating a first new intermediate frame based on the first refined optical flow map; and generating a second new intermediate frame based on the second refined optical flow map. ### Claim 2 (depends on 1) The method of claim 1, further comprising: receiving a video sequence of a video, comprising: a beginning frame of the video sequence, the beginning frame corresponding to a beginning time of the video sequence; and an ending frame of the video sequence, the ending frame corresponding to an ending time of the video sequence; receiving two additional frames of the video corresponding to two additional times, each additional time being either before the beginning time or after the ending time of the video sequence; generating at least one initial optical flow map corresponding to estimated optical flow between the beginning frame and the ending frame; and processing the at least one initial optical flow map to generate the plurality of intermediate optical flow maps, the plurality of intermediate optical flow maps corresponding to a plurality of intermediate times between the beginning time and ending time of the video sequence, by applying a motion model to the at least one initial optical flow map. ### Claim 3 (depends on 2) The method of claim 2, wherein the motion model is cubic or higher in order. ### Claim 4 (depends on 1) The method of claim 1, further comprising: processing the plurality of optical flow map to generate at least one additional refined optical flow map, wherein the first refined optical flow map, the second refined optical flow map, and the at least one additional refined optical flow map constitute a plurality of refined optical flow maps corresponding to the plurality of times; and generating at least one additional new intermediate frame based on the at least one additional refined optical flow map, wherein the first new intermediate frame, the second new intermediate frame, and the at least one additional new intermediate frame constitute a plurality of new intermediate frames corresponding to the plurality of times. ### Claim 5 (depends on 4) The method of claim 4, further comprising: obtaining the plurality of new intermediate frames; and refining the first new intermediate frame and the second new intermediate frame by: generating a first refined frame corresponding to a first time proximal to the beginning time or ending time, based on the plurality of new intermediate frames; and generating a second refined frame corresponding to a second time distal from both the beginning time and ending time, based on the plurality of new intermediate frames and the first refined frame. ### Claim 6 (depends on 5) The method of claim 5, further comprising refining the plurality of new intermediate frames to generate a plurality of refined new intermediate frames based on the plurality of new intermediate frames. ### Claim 7 (depends on 1) The method of claim 1, further comprising generating the plurality of intermediate optical flow maps using an optical flow model that is trained using a supervised learning algorithm, a set of training data, and a relaxed loss function configured to allow a predetermined amount of error in the estimated optical flow during training of the optical flow refinement model. ### Claim 8 (depends on 7) The method of claim 7, wherein the relaxed loss function comprises an adversarial loss function generated by a generative adversarial network. ### Claim 9 (depends on 4) The method of claim 4, wherein the at least one additional refined optical flow map is generated using a pixel refinement model that is trained using an adversarial loss function generated by a generative adversarial network. ### Claim 10 (depends on 1) The method of claim 1, wherein processing the plurality of optical flow maps to generate a first refined optical flow map and processing the plurality of optical flow maps and the first refined intermediate optical flow map to generate a second refined optical flow map is performed using an optical flow refinement model that is trained using an adversarial loss function generated by a generative adversarial network. ### Claim 11 A computing system comprising: a processor; and a memory storing processor-executable instructions thereon, wherein the instructions, when executed by the processor, cause the computing system to: obtain a plurality of optical flow maps corresponding to a plurality of times between a beginning time and an ending time; processing the plurality of optical flow maps to generate a first refined optical flow map corresponding to a first intermediate time proximal to the beginning time or ending time; processing the plurality of optical flow maps and the first refined optical flow map to generate a second refined optical flow map corresponding to a second intermediate time distal from both the beginning time and ending time; generate a first new intermediate frame based on the first refined optical flow map; and generate a second new intermediate frame based on the second refined optical flow map. ### Claim 12 (depends on 11) The computing system of claim 11, wherein the instructions, when executed by the processor, further cause the computing system to: receive a video sequence of a video, comprising: a beginning frame of the video sequence, the beginning frame corresponding to a beginning time of the video sequence; and an ending frame of the video sequence, the ending frame corresponding to an ending time of the video sequence; receive two additional frames of the video corresponding to two additional times, each additional time being either before the beginning time or after the ending time of the video sequence; generate at least one initial optical flow map corresponding to estimated optical flow between the beginning frame and the ending frame; and process the at least one initial optical flow map to generate the plurality of intermediate optical flow maps, the plurality of intermediate optical flow maps corresponding to a plurality of intermediate times between the beginning time and ending time of the video sequence, by applying a motion model to the at least one initial optical flow map. ### Claim 13 (depends on 12) The computing system of claim 12, wherein the motion model is cubic or higher in order. ### Claim 14 (depends on 11) The computing system of claim 11, wherein the instructions, when executed by the processor, further cause the computing system to: Process the plurality of optical flow maps to generate at least one additional refined optical flow map, wherein the first refined optical flow map, the second refined optical flow map, and the at least one additional refined optical flow map constitute a plurality of refined optical flow maps corresponding to the plurality of times; and generate at least one additional new intermediate frame based on the at least one additional refined optical flow map, wherein the first new intermediate frame, the second new intermediate frame, and the at least one additional new intermediate frame constitute a plurality of new intermediate frames corresponding to the plurality of times. ### Claim 15 (depends on 14) The computing system of claim 14, wherein the instructions, when executed by the processor, further cause the computing system to: obtain the plurality of new intermediate frames; and refine the first new intermediate frame and the second new intermediate frame by: generating a first refined frame corresponding to a first time proximal to the beginning time or ending time, based on the plurality of new intermediate frames; and generating a second refined frame corresponding to a second time distal from both the beginning time and ending time, based on the plurality of new intermediate frames and the first refined frame. ### Claim 16 A multi-frame video frame interpolation system, comprising: an optical flow refinement module configured to: obtain a plurality of optical flow maps corresponding to a plurality of times between a beginning time and an ending time; process the plurality of optical flow maps to generate a first refined optical flow map corresponding to a first intermediate time proximal to the beginning time or ending time; process the plurality of optical flow maps and the first refined intermediate optical flow map to generate a second refined optical flow map corresponding to a second intermediate time distal from both the beginning time and ending time; generate a first new intermediate frame based on the first refined optical flow map; and generate a second new intermediate frame based on the second refined optical flow map. ### Claim 17 (depends on 16) The multi-frame video frame interpolation system of claim 16, further comprising: an optical flow estimation module configured to: receive a video sequence of a video, comprising: a beginning frame of the video sequence, the beginning frame corresponding to a beginning time of the video sequence; and an ending frame of the video sequence, the ending frame corresponding to an ending time of the video sequence; receive two additional frames of the video corresponding to two additional times, each additional time being either before the beginning time or after the ending time of the video sequence; and generate at least one initial optical flow map corresponding to estimated optical flow between the beginning frame and the ending frame; and a motion modeling module configured to process the at least one initial optical flow map to generate the plurality of intermediate optical flow maps, the plurality of intermediate optical flow maps corresponding to a plurality of intermediate times between the beginning time and ending time of the video sequence, by applying a motion model to the at least one initial optical flow map. ### Claim 18 (depends on 17) The multi-frame video frame interpolation system of claim 17, wherein the motion modeling module is configured to generate the plurality of intermediate optical flow maps using a motion model that is cubic or higher in order. ### Claim 19 (depends on 16) The multi-frame video frame interpolation system of claim 16, wherein the optical flow refinement module is further configured to: process the plurality of optical flow maps to generate at least one additional refined optical flow ma, wherein the first refined optical flow map, the second refined optical flow map, and the at least one additional refined optical flow map constitute a plurality of refined optical flow maps corresponding to the plurality of times; and generate at least one additional new intermediate frame based on the at least one additional refined optical flow map, wherein the first new intermediate frame, the second new intermediate frame, and the at least one additional new intermediate frame constitute a plurality of new intermediate frames corresponding to the plurality of times. ### Claim 20 (depends on 19) The multi-frame video frame interpolation system of claim 19, further comprising a pixel refinement module configured to: obtain the plurality of new intermediate frames; and refine the first new intermediate frame and the second new intermediate frame by: generating a first refined frame corresponding to a first time proximal to the beginning time or ending time, based on the plurality of new intermediate frames; and generating a second refined frame corresponding to a second time distal from both the beginning time and ending time, based on the plurality of new intermediate frames and the first refined frame.