# US 12,120,359 B2 · Machine learning model-based video compression

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,120,359 B2
- Document ID: US12120359B2
- Status: Granted
- Kind: B2
- Application number: 17/704,692
- Filed: 2022-03-25
- Published: 2022-10-13
- Granted: 2024-10-15
- Earliest claimed: 2021-04-08
- Assignees: Disney Enterprises, Inc.; ETH Zürich (Eidgenössische Technische Hochschule Zürich)
- Applicants: Disney Enterprises, Inc.; ETH Zürich (EIDGENÖSSISCHE TECHNISCHE HOCHSCHULE ZÜRICH)
- Inventors: Abdelaziz Djelouah; Leonhard Markus Helminger; Roberto Gerson De Albuquerque Azevedo; Scott Labrozzi; Christopher Richard Schroers; Yuanyi Xue
- Web page: https://patentgrep.com/patent/US12120359B2

## CPC

- H04N 19/503: involving temporal prediction (adaptive coding with adaptive selection between spatial and temporal predictive coding H04N19/107; adaptive coding with adaptive selection among a plurality of temporal predictive coding modes H04N19/109)
- H04N 19/89: involving methods or arrangements for detection of transmission errors at the decoder
- G06N 3/045: Combinations of networks
- G06N 3/0455: Auto-encoder networks; Encoder-decoder networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/0475: Generative networks
- G06N 3/088: Non-supervised learning, e.g. competitive learning
- G06N 3/09: Supervised learning
- G06N 3/094: Adversarial learning
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

A system processing hardware executes a machine learning (ML) model-based video compression encoder to receive uncompressed video content and corresponding motion compensated video content, compare the uncompressed and motion compensated video content to identify an image space residual, transform the image space residual to a latent space representation of the uncompressed video content, and transform, using a trained image compression ML model, the motion compensated video content to a latent space representation of the motion compensated video content. The ML model-based video compression encoder further encodes the latent space representation of the image space residual to produce an encoded latent residual, encodes, using the trained image compression ML model, the latent space representation of the motion compensated video content to produce an encoded latent video content, and generates, using the encoded latent residual and the encoded latent video content, a compressed video content corresponding to the uncompressed video content.

## Claims

### Claim 1

A system comprising:
a computing platform including a processing hardware and a system memory storing a machine learning (ML) model-based video compression encoder and a trained image compression ML model;
the processing hardware configured to execute the ML model-based video compression encoder to:
receive an uncompressed video content and a motion compensated video content corresponding to the uncompressed video content;
compare the uncompressed video content with the motion compensated video content to identify an image space residual corresponding to the uncompressed video content;
transform the image space residual to a latent space representation of the image space residual;
receive, using the trained image compression ML model, the motion compensated video content;
transform, using the trained image compression ML model, the motion compensated video content to a latent space representation of the motion compensated video content;
encode the latent space representation of the image space residual to produce an encoded latent residual;
encode, using the trained image compression ML model, the latent space representation of the motion compensated video content to produce an encoded latent video content; and
generate, using the encoded latent residual and the encoded latent video content, a compressed video content corresponding to the uncompressed video content.

### Claim 2 (depends on 1)

The system of claim 1, wherein the encoded latent residual and the encoded latent video content are produced in parallel.

### Claim 3 (depends on 1)

The system of claim 1, wherein the processing hardware is configured to execute the ML model-based video compression encoder to generate the compressed video content corresponding to the uncompressed video content based on a difference between the encoded latent residual and the encoded latent video content.

### Claim 4 (depends on 1)

The system of claim 1, wherein the trained image compression ML model comprises a trained artificial neural network (NN).

### Claim 5 (depends on 4)

The system of claim 4, wherein the trained NN is trained using an objective function including an adversarial loss.

### Claim 6 (depends on 4)

The system of claim 4, wherein the trained NN comprises a generative adversarial network (GAN).

### Claim 7

A method for use by a system including a computing platform having a processing hardware and a system memory storing a machine learning (ML) model-based video compression encoder and a trained image compression ML model, the method comprising:
receiving, by the ML model-based video compression encoder executed by the processing hardware, an uncompressed video content and a motion compensated video content corresponding to the uncompressed video content;
comparing, by the ML model-based video compression encoder executed by the processing hardware, the uncompressed video content with the motion compensated video content, thereby identifying an image space residual corresponding to the uncompressed video content;
transforming, by the ML model-based video compression encoder executed by the processing hardware, the image space residual to a latent space representation of the image space residual;
receiving, by the trained image compression ML model executed by the processing hardware, the motion compensated video content;
transforming, by the trained image compression ML model executed by the processing hardware, the motion compensated video content to a latent space representation of the motion compensated video content;
encoding, by the ML model-based video compression encoder executed by the processing hardware, the latent space representation of the image space residual to produce an encoded latent residual;
encoding, by the trained image compression ML model executed by the processing hardware, the latent space representation of the motion compensated video content to produce an encoded video content; and
generating, by the ML model-based video compression encoder executed by the processing hardware and using the encoded latent residual and the encoded latent video content, a compressed video content corresponding to the uncompressed video content.

### Claim 8 (depends on 7)

The method of claim 7, wherein the encoded latent residual and the encoded latent video content are produced in parallel.

### Claim 9 (depends on 7)

The method of claim 7, wherein the compressed video content corresponding to the uncompressed video content is generated based on a difference between the encoded latent residual and the encoded latent video content.

### Claim 10 (depends on 7)

The method of claim 7, wherein the trained image compression ML model comprises a trained artificial neural network (NN).

### Claim 11 (depends on 10)

The method of claim 10, wherein the trained NN is trained using an objective function including an adversarial loss.

### Claim 12 (depends on 7)

The method of claim 7, wherein the trained NN comprises a generative adversarial network (GAN).

### Claim 13

A system comprising:
a computing platform including a processing hardware and a system memory storing a machine learning (ML) model-based video compression encoder and a trained image compression ML model;
the processing hardware configured to execute the ML model-based video compression encoder to:
receive, using the trained image compression ML model, an uncompressed video content and a motion compensated video content corresponding to the uncompressed video content;
transform, using the trained image compression ML model, the uncompressed video content to a first latent space representation of the uncompressed video content;
transform, using the trained image compression ML model, the motion compensated video content to a second latent space representation of the uncompressed video content; and
generate a bitstream for transmitting a compressed video content corresponding to the uncompressed video content based on the first latent space representation and the second latent space representation.

### Claim 14 (depends on 13)

The system of claim 13, wherein the processing hardware is further configured to execute the ML model-based video compression encoder to:
determine, using the first latent space representation and the second latent space representation, a latent space residual.

### Claim 15 (depends on 14)

The system of claim 14, wherein the processing hardware is further configured to execute the ML model-based video compression encoder to:
generate the bitstream for transmitting the compressed video content corresponding to the uncompressed video content using the latent space residual.

### Claim 16 (depends on 14)

The system of claim 14, wherein the latent space residual is based on a difference between the first latent space representation and the second latent space representation.

### Claim 17 (depends on 13)

The system of claim 13, wherein the transformation of the uncompressed video content to the first latent space representation, and the transformation of the motion compensated video content to the second latent space representation, are performed in parallel.

### Claim 18 (depends on 13)

The system of claim 13, wherein the trained image compression ML model comprises a trained artificial neural network (NN).

### Claim 19 (depends on 13)

The system of claim 13, wherein the trained NN is trained using an objective function including an adversarial loss.

### Claim 20 (depends on 13)

The system of claim 13, wherein the trained NN comprises a generative adversarial network (GAN).