# US 11,122,238 B1 · Frame interpolation with multi-scale deep loss functions and generative adversarial networks

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,122,238 B1
- Document ID: US11122238B1
- Status: Granted
- Kind: B1
- Application number: 16/183,311
- Filed: 2018-11-07
- Granted: 2021-09-14
- Earliest claimed: 2017-11-07
- Assignees: Twitter, Inc.
- Applicants: Twitter, Inc.
- Inventors: Joost van Amersfoort; Wenzhe Shi; Jose Caballero; Alfredo Alejandro Acosta Diaz; Francisco Massa; Johannes Totz; Zehan Wang
- Web page: https://patentgrep.com/patent/US11122238B1

## CPC

- G06T 3/4053: based on super-resolution, i.e. the output image resolution being higher than the sensor resolution
- H04N 7/0137: {dependent on presence/absence of motion, e.g. of motion zones (H04N7/014 takes precedence; movement detection in television signals H04N5/144)}
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/047: Probabilistic or stochastic networks
- G06N 3/08: Learning methods
- G06N 3/088: Non-supervised learning, e.g. competitive learning
- G06N 3/09: Supervised learning
- G06N 3/094: Adversarial learning
- G06T 3/4007: based on interpolation, e.g. bilinear interpolation (image demosaicing G06T3/4015; edge-driven or edge-based scaling G06T3/403)
- H04N 7/0127: {by changing the field or frame frequency of the incoming video signal, e.g. frame rate converter}
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

A method includes selecting two or more frames from a plurality of frames of a video, downscaling the two or more frames, estimating a flow data based on an optical flow associated with the downscaled two or more frames, upscaling the flow data, generating a refined flow data based on the upscaled flow data and the downscaled two or more frames, upscaling the refined flow data, and synthesizing an image based on the upscaled refined flow data and the two or more frames.

## Claims

### Claim 1

A method comprising: selecting two or more frames from a plurality of frames of a video; downscaling the two or more frames; estimating flow data based on the downscaled two or more frames using a trained first neural network, wherein the neural network is a trained optical flow model that progressively applies and refines flow fields; upscaling the estimated flow data; generating, using a second neural network that is trained to estimate optical flow fields, refined flow data based on the upscaled estimated flow data and the downscaled two or more frames, the upscaled estimated flow data and the downscaled two or more frames having a same scaling factor; upscaling the refined flow data; synthesizing an image based on the upscaled refined flow data and the two or more frames; and inserting the image as an interpolated frame into the plurality of frames of the video.

### Claim 2 (depends on 1)

The method of claim 1, wherein the optical flow is estimated based on a displacement that each pixel of the downscaled two or more frames undergoes from one frame to the next.

### Claim 3 (depends on 1)

The method of claim 1, wherein
the downscaling of the two or more frames includes a plurality of downscaling operations using a scaling factor, and
the upscaling of the estimated flow data includes upscaling by the scaling factor.

### Claim 4 (depends on 1)

The method of claim 1, wherein
the downscaling of the two or more frames includes:
applying a first downscaling using a scaling factor,
applying a second downscaling using the scaling factor,
applying a third downscaling using the scaling factor, and
the upscaling of the estimated flow data includes upscaling the estimated flow data by the scaling factor,
the generating of the refined flow data includes generating a first refined flow data, and
the synthesizing of the image includes synthesizing a first image, and the method further comprises:
upscaling the first refined flow data;
generating a second refined flow data based on the upscaled first refined flow data and the first downscaled two or more frames;
upscaling the second refined flow data; and
synthesizing a second image based on the upscaled second refined flow data and the two or more frames.

### Claim 5 (depends on 1)

The method of claim 1, further comprising:
synthesizing a second image based on the upscaled estimated flow data and the two or more frames.

### Claim 6 (depends on 1)

The method of claim 1, wherein the synthesizing of the image includes synthesizing a first image, the method further comprising:
synthesizing a second image based on the upscaled estimated flow data and the two or more frames;
synthesizing a third image based on further upscaled, further refined flow data and the two or more frames; and
generating a frame using the first image, the second image and the third image.

### Claim 7 (depends on 1)

The method of claim 1, wherein
the estimating of the flow data includes using convolutional neural network (CNN) having a trained loss function,
the generating of the refined flow data includes using the CNN having the trained loss function, and
the synthesizing of the image includes using the CNN having the trained loss function.

### Claim 8 (depends on 7)

The method of claim 7, wherein the loss function is trained using a Generative Adversarial Network.

### Claim 9

A system comprising:
a processor; and
a memory including code segments that when executed by the processor cause the processor to:
select two or more frames from a plurality of frames of a video;
downscale the two or more frames,
estimate flow data based on the downscaled two or more frames using a first neural network, wherein the neural network is a trained optical flow model that progressively applies and refines flow fields,
upscale the estimated flow data,
generate, using a second neural network that is trained to estimate optical flow fields, refined flow data based on the upscaled estimated flow data and the downscaled two or more frames, the upscaled estimated flow data and the downscaled two or more frames having a same scaling factor,
upscale the refined flow data, and
synthesize an image based on the upscaled refined flow data and the two or more frames; and
insert the image between the two or more frames as an interpolated frame in the video.

### Claim 10 (depends on 9)

The system of claim 9, wherein the optical flow is estimated based on a displacement that each pixel of the downscaled two or more frames undergoes from one frame to the next.

### Claim 11 (depends on 9)

The system of claim 9, wherein
the downscaling of the two or more frames includes a plurality of downscaling operations using a scaling factor, and
the upscaling of the estimated flow data includes upscaling by the scaling factor.

### Claim 12 (depends on 9)

The system of claim 9, wherein
the downscaling of the two or more frames includes:
applying a first downscaling using a scaling factor,
applying a second downscaling using the scaling factor,
applying a third downscaling using the scaling factor, and
the upscaling of the estimated flow data includes upscaling the estimated flow data by the scaling factor,
the generating of the refined flow data includes generating a first refined flow data, and
the synthesizing of the image is synthesizing a first image, and the code segments that when executed by the processor further cause the processor to:
upscale the first refined flow data;
generate a second refined flow data based on the upscaled first refined flow data and the first downscaled two or more frames;
upscale the second refined flow data; and
synthesize a second image based on the upscaled second refined flow data and the two or more frames.

### Claim 13 (depends on 9)

The system of claim 9, further comprising: synthesizing a second image based on the upscaled estimated flow data and the two or more frames.

### Claim 14 (depends on 9)

The system of claim 9, wherein the synthesizing of the image is synthesizing a first image, the code segments that when executed by the processor further cause the processor to:
synthesize a second image based on the upscaled estimated flow data and the two or more frames;
synthesize a third image based on further upscaled, further refined flow data and the two or more frames; and
generate a frame using the first image, the second image and the third image.

### Claim 15 (depends on 9)

The system of claim 9, wherein
the estimating of the flow data includes using convolutional neural network (CNN) having a trained loss function,
the generating of the refined flow data includes using the CNN having the trained loss function, and
the synthesizing of the image includes using the CNN having the trained loss function.

### Claim 16 (depends on 15)

The system of claim 15, wherein the loss function is trained using a Generative Adversarial Network.

### Claim 17

A non-transitory computer readable storage medium including instructions that when executed by at least one processor, cause the at least one processor to:
select two or more frames from a plurality of frames of a video;
downscale the two or more frames;
estimate flow data based on the downscaled two or more frames using a first neural network, wherein the neural network is a trained optical flow model that progressively applies and refines flow fields;
upscale the estimated flow data;
generate, using a second neural network that is trained to estimate optical flow fields, refined flow data based on the upscaled estimated flow data and the downscaled two or more frames, the upscaled estimated flow data and the downscaled two or more frames having a same scaling factor;
upscale the refined flow data;
synthesize an image based on the upscaled refined flow data and the two or more frames; and
insert the image as an interpolated frame into the plurality of frames of the video.

### Claim 18 (depends on 17)

The computer readable storage medium of claim 17, wherein
the estimating of the flow data includes using convolutional neural network (CNN) having a trained loss function,
the generating of the refined flow data includes using the CNN having the trained loss function, and
the synthesizing of the image includes using the CNN having the trained loss function.

### Claim 19 (depends on 18)

The computer readable storage medium of claim 18, wherein the loss function is trained using a Generative Adversarial Network.

### Claim 20 (depends on 17)

The computer readable storage medium of claim 17, wherein the optical flow is estimated based on a displacement that each pixel of the downscaled two or more frames undergoes from one frame to the next.