# US 2025/0168367 A1 · Tunable Hybrid Neural Video Representations

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2025/0168367 A1
- Document ID: US20250168367A1
- Status: Published application
- Kind: A1
- Application number: 18/920,734
- Filed: 2024-10-18
- Published: 2025-05-22
- Earliest claimed: 2023-11-16
- Assignees: Disney Enterprises, Inc.; ETH Zürich (Eidgenössische Technische Hochschule Zürich)
- Applicants: Disney Enterprises, Inc.; Eth Zurich (Eidgenossische Technische Hochschule Zurich)
- Inventors: Roberto Gerson de Albuquerque Azevedo; Christopher Richard Schroers; Scott Labrozzi; Jens Eirik Saethre; Yuanyl Xue
- Web page: https://patentgrep.com/patent/US20250168367A1

## CPC

- H04N 19/189: characterised by the adaptation method, adaptation tool or adaptation type used for the adaptive coding
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06N 3/088: Non-supervised learning, e.g. competitive learning
- G06V 10/82: using neural networks
- G06V 20/46: {Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames}
- H04N 19/136: Incoming video signal characteristics or properties
- H04N 19/172: the region being a picture, frame or field
- H04N 19/177: the unit being a group of pictures [GOP]
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

A system includes a tunable neural network-based video encoder configured to receive a video sequence including multiple video frames, generate a frame-specific embedding of a first video frame of the multiple video frames, and identify one or more group-of-pictures (GOP) features of a subset of the multiple video frames, the subset of the including the first video frame. The tunable neural network-based video encoder is further configured to combine the frame-specific embedding of the first video frame and the one or more GOP features of the first plurality of the plurality of video frames to provide a latent feature corresponding to a compressed version of the first video frame.

## Claims

### Claim 1

A system comprising:
a tunable neural network-based video encoder configured to:
receive a video sequence including a plurality of video frames;
generate a frame-specific embedding of a first video frame of the plurality of video frames;
identify one or more group-of-pictures (GOP) features of a first plurality of the plurality of video frames, the first plurality of the plurality of video frames including the first video frame; and
combine the frame-specific embedding of the first video frame and the one or more GOP features of the first plurality of the plurality of video frames to provide a latent feature corresponding to a compressed version of the first video frame.

### Claim 2 (depends on 1)

The system of claim 1, wherein the one or more GOP features of the first plurality of the plurality of video frames are identified using a multi-resolution feature grid of the tunable neural network-based video encoder.

### Claim 3 (depends on 1)

The system of claim 1, wherein tunable neural network-based video encoder configured to:
extract, from the first plurality of the plurality of video frames, one or more temporal features of the first plurality of the plurality of video frames;
wherein the frame-specific embedding is generated based on a combination with the one or more temporal features.

### Claim 4 (depends on 1)

The system of claim 1, wherein the latent feature comprises a weighted combination of the frame-specific embedding and the one or more GOP features.

### Claim 5 (depends on 4)

The system of claim 4, wherein at least one of a first weight applied to the frame-specific embedding or a second weight applied to the one or more GOP features is determined by the tunable neural network-based video encoder.

### Claim 6 (depends on 1)

The system of claim 1, wherein the tunable neural network-based video encoder is trained using an optimized loss function including a distortion loss and an entropy loss.

### Claim 7 (depends on 6)

The system of claim 6, wherein the distortion loss and the entropy loss are optimized jointly.

### Claim 8 (depends on 1)

The system of claim 1, wherein the first video frame comprises a sub-sequence of the first plurality of the plurality of video frames, and wherein the frame-specific embedding comprises a plurality of frame-specific embeddings corresponding respectively to each of the sub-sequence of the first plurality of the plurality of video frames, and the latent feature is one of a plurality of latent features each corresponding respectively to a compressed version of each of the sub-sequence of the first plurality of the plurality of video frames.

### Claim 9

A system comprising:
a neural network-based video decoder configured to:
receive a latent feature corresponding to a compressed version of a first video frame of a first plurality of a plurality of video frames included in a video sequence, the latent feature being a combination of a frame-specific embedding of the first video frame with one or more group-of-pictures (GOP) features of the first plurality of the plurality of video frames; and
decode the latent feature to provide an uncompressed video frame corresponding to the first video frame.

### Claim 10 (depends on 9)

The system of claim 9, wherein the frame-specific embedding is generated based on a combination with one or more temporal features extracted from the first plurality of the plurality of video frames.

### Claim 11 (depends on 9)

The system of claim 9, wherein the latent feature comprises a weighted combination of the frame-specific embedding and the one or more GOP features.

### Claim 12 (depends on 9)

The system of claim 9, wherein the first video frame comprises a sub-sequence of the first plurality of the plurality of video frames, and wherein the frame-specific embedding comprises a plurality of frame-specific embeddings corresponding respectively to each of the sub-sequence of the first plurality of the plurality of video frames, and the latent feature is one of a plurality of latent features each corresponding respectively to a compressed version of each of the sub-sequence of the first plurality of the plurality of video frames.

### Claim 13

A method for execution by a tunable neural network-based video encoder, the method comprising:
receiving a video sequence including a plurality of video frames;
generating a frame-specific embedding of a first video frame of the plurality of video frames;
identifying one or more group-of-pictures (GOP) features of a first plurality of the plurality of video frames, the first plurality of the plurality of video frames including the first video frame; and
combining the frame-specific embedding of the first video frame and the one or more GOP features of the first plurality of the plurality of video frames to provide a latent feature corresponding to a compressed version of the first video frame.

### Claim 14 (depends on 13)

The method of claim 13, wherein the one or more GOP features of the first plurality of the plurality of video frames are identified using a multi-resolution feature grid of the tunable neural network-based video encoder.

### Claim 15 (depends on 13)

The method of claim 13, further comprising:
extracting from the first plurality of the plurality of video frames one or more temporal features of the first plurality of the plurality of video frames;
wherein the frame-specific embedding is generated based on a combination with the one or more temporal features.

### Claim 16 (depends on 13)

The method of claim 13, wherein the latent feature comprises a weighted combination of the frame-specific embedding and the one or more GOP features.

### Claim 17 (depends on 16)

The method of claim 16, further comprising:
determining at least one of a first weight applied to the frame-specific embedding or a second weight applied to the one or more GOP features.

### Claim 18 (depends on 13)

The method of claim 13, wherein the tunable neural network-based video encoder is trained using an optimized loss function including a distortion loss and an entropy loss.

### Claim 19 (depends on 18)

The method of claim 18, wherein the distortion loss and the entropy loss are optimized jointly.

### Claim 20 (depends on 13)

The method of claim 13, wherein the first video frame comprises a sub-sequence of the first plurality of the plurality of video frames, and wherein the frame-specific embedding comprises a plurality of frame-specific embeddings corresponding respectively to each of the sub-sequence of the first plurality of the plurality of video frames, and the latent feature is one of a plurality of latent features each corresponding respectively to a compressed version of each of the sub-sequence of the first plurality of the plurality of video frames.
