# US 2024/0291994 A1 · Method and data processing system for lossy image or video encoding, transmission and decoding

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2024/0291994 A1
- Document ID: US20240291994A1
- Status: Published application
- Kind: A1
- Application number: 18/458,511
- Filed: 2023-08-30
- Published: 2024-08-29
- Earliest claimed: 2023-02-19
- Assignees: Deep Render Ltd.
- Applicants: Deep Render Ltd
- Inventors: Bilal ABBASI; Sebastjan CIZEL; Chris FINLAY; Christian ETMANN; Arsalan ZAFAR
- Web page: https://patentgrep.com/patent/US20240291994A1

## CPC

- G06T 9/002: {using neural networks}
- G06T 3/18: Image warping, e.g. rearranging pixels individually
- H04N 19/137: Motion inside a coding unit, e.g. average field, frame or block difference
- H04N 19/172: the region being a picture, frame or field
- H04N 19/196: being specially adapted for the computation of encoding parameters, e.g. by averaging previously computed encoding parameters (processing of motion vectors H04N19/513)
- H04N 19/91: Entropy coding, e.g. variable length coding [VLC] or arithmetic coding
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

A method for lossy video encoding, transmission and decoding, the method comprising the steps of: receiving a first frame and a second frame at a first computer system; determining a first flow between the first frame and the second frame; determining a second flow based on the first frame and the second frame; encoding an input based on the first flow and the second flow using a first trained neural network to produce a latent representation; transmitting the latent representation to a second computer system; decoding the latent representation using a second trained neural network to produce an output flow; and using the output flow to obtain an output frame, wherein the output frame is an approximation of the second frame.

## Claims

### Claim 1

A method for lossy video encoding, transmission and decoding, the method comprising the steps of:
receiving a first frame and a second frame at a first computer system;
determining a first flow between the first frame and the second frame;
determining a second flow based on the first frame and the second frame;
encoding an input based on the first flow and the second flow using a first trained neural network to produce a latent representation;
transmitting the latent representation to a second computer system;
decoding the latent representation using a second trained neural network to produce an output flow; and
using the output flow to obtain an output frame, wherein the output frame is an approximation of the second frame;
wherein each of the first flow and the second flow define a transformation of a part of an image from a first position in the first frame to a second position in the second frame.

### Claim 2 (depends on 1)

The method of claim 1, further comprising the step of:
transforming the first frame using the first flow to obtain a warped frame;
wherein the second flow is determined between the warped frame and the second frame.

### Claim 3 (depends on 2)

The method of claim 2, wherein the input based on the first flow and the second flow is the second flow.

### Claim 4 (depends on 2)

The method of claim 2, wherein the step of obtaining the output frame comprises applying the output flow to a frame corresponding to the first frame to obtain an intermediate frame.

### Claim 5 (depends on 4)

The method of claim 4, wherein the step of obtaining the output frame comprises applying the first flow to the intermediate frame to obtain the output frame.

### Claim 6 (depends on 1)

The method of claim 1, wherein the second flow is a point-wise flow; and
the input based on the first flow and the second flow is the first flow subtracted from the second flow.

### Claim 7 (depends on 6)

The method of claim 6, wherein the step of obtaining the output frame comprises adding the first flow to the output flow to obtain a reconstructed flow.

### Claim 8 (depends on 7)

The method of claim 7, wherein the step of obtaining the output frame comprises applying the reconstructed flow to a frame corresponding to the first frame to obtain the output frame.

### Claim 9 (depends on 2)

The method of claim 2, wherein the input based on the first flow and the second flow is the first flow and the second flow.

### Claim 10 (depends on 9)

The method of claim 9, wherein the step of obtaining the output frame comprises applying the output flow to a frame corresponding to the first frame to obtain the output frame.

### Claim 11 (depends on 1)

The method of claim 1, further comprising the step of dividing each of the first frame and the second frame into a plurality of corresponding sub-frames; and
a first flow is determined for each of corresponding sub-frames.

### Claim 12 (depends on 1)

The method of claim 1, wherein the first flow is a linear function.

### Claim 13 (depends on 12)

The method of claim 12, wherein the first flow is determined by one of the following methods: a neural network that receives the first frame and the second frame as input and outputs one or more parameters of a matrix defining the first flow, a linear-least squares fit, a Fourier-Mellin transformation and an optimisation that minimizes the photometric error between the first frame and the second frame.

### Claim 14 (depends on 1)

The method of claim 1, wherein the first flow is a non-linear function.

### Claim 15 (depends on 14)

The method of claim 14, wherein the first flow comprises a frame transformation.

### Claim 16 (depends on 14)

The method of claim 14, wherein the non-linear function is a trained neural network.

### Claim 17 (depends on 1)

The method of claim 1, wherein the first flow is a global flow.

### Claim 18 (depends on 1)

The method of claim 1, wherein the second flow is a local flow.

### Claim 19

A method of training one or more neural networks, the one or more neural networks being for use in lossy video encoding, transmission and decoding, the method comprising the steps of:
receiving a first frame and a second frame at a first computer system; determining a first flow between the first frame and the second frame;
determining a second flow based on the first frame and the second frame;
encoding an input based on the first flow and the second flow using a first neural network to produce a latent representation;
transmitting the latent representation to a second computer system;
decoding the latent representation using a second neural network to produce an output flow;
using the output flow to obtain an output frame, wherein the output frame is an approximation of the second frame;
evaluating a function based on a difference between the output frame and the second frame;
updating the parameters of the first neural network and the second neural network based on the evaluated function; and
repeating the above steps using a first set of input frames to produce a first trained neural network and a second trained neural network;
wherein each of the first flow and the second flow define a transformation of a part of an image from a first position in the first frame to a second position in the second frame.

### Claim 20 (depends on 1)

A data processing system configured to perform the method of claim 1.
