# US 2023/0267706 A1 · Video remastering via deep learning

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2023/0267706 A1
- Document ID: US20230267706A1
- Status: Published application
- Kind: A1
- Application number: 18/172,201
- Filed: 2023-02-21
- Published: 2023-08-24
- Earliest claimed: 2022-02-21
- Assignees: Disney Enterprises, Inc.
- Applicants: Disney Enterprises, Inc.
- Inventors: Abdelaziz DJELOUAH; Shinobu HATTORI; Christopher Richard SCHROERS; Andrew John WAHLQUIST
- Web page: https://patentgrep.com/patent/US20230267706A1

## CPC

- G06T 3/4092: Image resolution transcoding, e.g. by using client-server architectures
- G06V 10/771: Feature selection, e.g. selecting representative features from a multi-dimensional feature space
- G06T 3/02: Affine transformations (for image registration G06T3/147; for image mosaicing G06T3/4038)
- G06T 3/40: Scaling of whole images or parts thereof, e.g. expanding or contracting
- G06T 3/4046: using neural networks
- G06T 5/10: using non-spatial domain filtering
- G06V 10/24: Aligning, centring, orientation detection or correction of the image
- G06V 10/62: relating to a temporal dimension, e.g. time-based feature extraction; Pattern tracking
- G06V 10/82: using neural networks
- G06V 20/46: {Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames}
- G06T 2207/20056: Discrete and fast Fourier transform, [DFT, FFT]
- G06T 2207/20081: Training; Learning
- G06T 2207/30168: Image quality inspection
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

One embodiment of the present invention sets forth a technique for performing remastering of video content. The technique includes determining a first input frame corresponding to a first frame included in a first video and a first target frame corresponding to a second frame included in a second video based on one or more alignments between the first frame and the second frame. The technique also includes executing a machine learning model to convert the first input frame into a first output frame. The technique further includes training the machine learning model based on one or more losses associated with the first output frame and the first target frame.

## Claims

### Claim 1

A computer-implemented method for performing remastering of video content, the computer-implemented method comprising:
determining a first input frame corresponding to a first frame included in a first video and a first target frame corresponding to a second frame included in a second video based on one or more alignments between the first frame and the second frame;
executing a machine learning model to convert the first input frame into a first output frame; and
training the machine learning model based on one or more losses associated with the first output frame and the first target frame.

### Claim 2 (depends on 1)

The computer-implemented method of claim 1, further comprising executing the machine learning model to convert a second input frame included in a third video into a second output frame.

### Claim 3 (depends on 2)

The computer-implemented method of claim 2, further comprising:
executing a discriminator model to generate a prediction associated with the second output frame; and
training the machine learning model based on one or more additional losses associated with the prediction.

### Claim 4 (depends on 2)

The computer-implemented method of claim 2, further comprising:
generating a first set of feature maps associated with the second output frame; and
training the machine learning model based on a perceptual loss computed between the first set of feature maps and a second set of feature maps associated with a second target frame included in a fourth video.

### Claim 5 (depends on 1)

The computer-implemented method of claim 1, wherein determining the first input frame comprises:
separating the first frame into a first set of scan lines and a second set of scan lines; and
generating the first input frame from the first set of scan lines.

### Claim 6 (depends on 1)

The computer-implemented method of claim 1, wherein determining the first input frame and the first target frame comprises determining at least one of the first input frame or the first target frame based on a temporal alignment between the first frame and the second frame.

### Claim 7 (depends on 1)

The computer-implemented method of claim 1, wherein determining the first input frame and the first target frame comprises generating at least one of the first input frame or the first target frame based on a geometric alignment between the first frame and the second frame.

### Claim 8 (depends on 1)

The computer-implemented method of claim 1, wherein determining the first input frame comprises resizing the first frame.

### Claim 9 (depends on 1)

The computer-implemented method of claim 1, wherein the one or more losses are computed based on a first Fast Fourier Transform (FFT) decomposition of the first output frame and a second FFT decomposition of the first target frame.

### Claim 10 (depends on 1)

The computer-implemented method of claim 1, wherein the one or more losses comprise an L1 loss.

### Claim 11

One or more non-transitory computer-readable media storing instructions that, when executed by one or more processors, cause the one or more processors to perform the steps of:
determining a first input frame corresponding to a first frame included in a first video and a first target frame corresponding to a second frame included in a second video based on one or more alignments between the first frame and the second frame;
executing a machine learning model to convert the first input frame into a first output frame; and
training the machine learning model based on one or more losses associated with the first output frame and the first target frame.

### Claim 12 (depends on 11)

The one or more non-transitory computer-readable media of claim 11, wherein the instructions further cause the one or more processors to perform the steps of:
executing the machine learning model to convert a second input frame included in a third video into a second output frame;
executing a discriminator model to generate a first prediction associated with the second output frame; and
training the machine learning model based on one or more additional losses associated with the first prediction.

### Claim 13 (depends on 12)

The one or more non-transitory computer-readable media of claim 12, wherein training the machine learning model based on the one or more losses and the one or more additional losses comprises:
performing a first training stage that trains the machine learning model based on the one or more losses; and
after the first training stage is complete, performing a second training stage based on a combination of the one or more losses and the one or more additional losses.

### Claim 14 (depends on 12)

The one or more non-transitory computer-readable media of claim 12, wherein the one or more additional losses comprise a weighted combination of a first loss that is computed based on the second output frame a second loss that is computed based on the first prediction.

### Claim 15 (depends on 12)

The one or more non-transitory computer-readable media of claim 12, wherein the instructions further cause the one or more processors to perform the step of training the discriminator model based on a loss that is computed from the first prediction and a second prediction generated by the discriminator model from a second target frame associated with the second input frame.

### Claim 16 (depends on 11)

The one or more non-transitory computer-readable media of claim 11, wherein training the machine learning model comprises:
generating a first set of feature maps associated with the first output frame and a second set of feature maps associated with the first target frame; and
training the machine learning model based on a perceptual loss computed between the first set of feature maps and the second set of feature maps.

### Claim 17 (depends on 11)

The one or more non-transitory computer-readable media of claim 11, wherein determining the first input frame and the first target frame comprises:
determining a temporal alignment between the first frame and the second frame; and
generating at least one of the first input frame or the first target frame based on a geometric alignment between the first frame and the second frame.

### Claim 18 (depends on 11)

The one or more non-transitory computer-readable media of claim 11, wherein determining the first input frame comprises:
determining an affine transformation based on a first set of spatial correspondences between the first frame and the second frame;
applying the affine transformation to the first frame to generate a transformed frame; and
generating the first input frame based on a second set of spatial correspondences between the transformed frame and the second frame.

### Claim 19 (depends on 11)

The one or more non-transitory computer-readable media of claim 11, wherein the one or more losses comprise an L1 loss that is computed between a first Fast Fourier Transform (FFT) decomposition of the first output frame and a second FFT decomposition of the first target frame.

### Claim 20

A system, comprising:
one or more memories that store instructions, and
one or more processors that are coupled to the one or more memories and, when executing the instructions, are configured to perform the steps of:
determining a first input frame corresponding to a first frame included in a first video and a first target frame corresponding to a second frame included in a second video based a temporal alignment between the first frame and the second frame and a geometric alignment between the first frame and the second frame;
executing a machine learning model to convert the first input frame into a first output frame; and
training the machine learning model based on one or more losses associated with the first output frame and the first target frame.

### Claim 21

A computer-implemented method for performing remastering of video content, the method comprising:
determining a first input frame corresponding to a first frame included in a first video, wherein the first input frame is associated with a first level of quality;
executing a machine learning model to convert the first input frame into a first output frame, wherein the first output frame is associated with a second level of quality that is higher than the first level of quality, and wherein the machine learning model is trained using a set of input frames that is associated with the first level of quality and a set of target frames that is temporally aligned with the set of input frames and associated with the second level of quality; and
generating a second video that includes the first output frame.
