# US 2025/0148753 A1 · Adaptive video compression using generative machine learning

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2025/0148753 A1
- Document ID: US20250148753A1
- Status: Published application
- Kind: A1
- Application number: 18/504,038
- Filed: 2023-11-07
- Published: 2025-05-08
- Earliest claimed: 2023-11-07
- Applicants: Microsoft Technology Licensing, LLC
- Inventors: Roei Shlomo MENASHOF; Oren ISTRIN; Itamar LATNIK
- Web page: https://patentgrep.com/patent/US20250148753A1

## CPC

- G06V 10/761: {Proximity, similarity or dissimilarity measures}
- H04N 21/26603: {for automatically generating descriptors from content, e.g. when it is not made available by its provider, using content analysis techniques}
- G06F 40/40: Processing or translation of natural language (natural language analysis G06F40/20; semantic analysis G06F40/30)
- G06N 3/045: Combinations of networks
- G06N 3/088: Non-supervised learning, e.g. competitive learning
- G06V 10/82: using neural networks
- G06V 20/46: {Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames}
- H04N 19/463: by compressing encoding parameters before transmission
- H04N 19/503: involving temporal prediction (adaptive coding with adaptive selection between spatial and temporal predictive coding H04N19/107; adaptive coding with adaptive selection among a plurality of temporal predictive coding modes H04N19/109)
- H04N 21/84: Generation or processing of descriptive data, e.g. content descriptors {(systems specially adapted for using meta-information in broadcast systems H04H60/73)}
- G06V 2201/07: Target detection
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

Various embodiments of the technology described herein relate to compression of video data, including selecting a pivot image from a video including a plurality of images and causing a first machine learning model to generate a descriptor of the pivot image, where the descriptor includes a language description associated with the pivot image. In one example, the pivot image and the descriptor are provided to a decoder for reconstruction of the video. In an embodiment, the decoder includes a generative machine learning model that takes as an input the pivot image and the descriptor. The decoder uses the pivot image to generate an image based at least in part on the descriptor. The image is combined with other images generated by the generative machine learning model to reconstruct the video.

## Claims

### Claim 1

A system comprising:
a memory component; and
a processing device coupled to the memory component, the processing device to perform operations comprising:
obtaining a video comprising a plurality of images;
selecting a pivot image from the plurality of images;
causing a first machine learning model to generate a descriptor based at least in part on the pivot image by at least providing the pivot image as an input to the first machine learning model, where the descriptor includes a language description of the pivot image; and
providing the pivot image and the descriptor to a decoder.

### Claim 2 (depends on 1)

The system of claim 1, wherein the pivot image depicts a conceptual element of the video.

### Claim 3 (depends on 1)

The system of claim 1, wherein selecting the pivot image further comprises selecting the pivot image from the plurality of images based on a second machine learning model detecting a change between two or more images of the plurality of images.

### Claim 4 (depends on 3)

The system of claim 3, wherein the change comprises a modification to an object depicted in the two or more images that is detected, by the second machine learning model.

### Claim 5 (depends on 3)

The system of claim 3, wherein the change comprises detecting, by the second machine learning model, an additional object relative to at least one image of the two or more images.

### Claim 6 (depends on 3)

The system of claim 3, wherein causing the first machine learning model to generate the descriptor further comprises prompting the first machine learning model to describe a conceptual element of the video relative to the pivot image and at least one other image of the plurality of images.

### Claim 7 (depends on 3)

The system of claim 3, wherein the processing device further performs operations causing, at the decoder, a third machine learning model to generate a reconstructed video by at least providing as a first input to the third machine learning model the pivot image and the descriptor, where the third machine learning model uses the pivot image and at least a portion of the descriptor to output a second plurality of images that are combined to generate the reconstructed video.

### Claim 8 (depends on 7)

The system of claim 7, wherein the first machine learning model comprises a large language model, the second machine learning model comprises a neural network, and the third machine learning model comprises a diffusion model.

### Claim 9

A non-transitory computer-readable medium storing executable instructions embodied thereon, that, when executed by a processing device, cause the processing device to perform operations comprising:
obtaining a pivot image from a video;
causing a machine learning model to generate a descriptor based at least in part on the pivot image, the descriptor providing a natural language description of the pivot image;
generating a compressed data object including the descriptor and the pivot image; and
providing the compressed data object to an endpoint over a network.

### Claim 10 (depends on 9)

The medium of claim 9, wherein the medium further stores executable instructions, that, cause the processing device to perform operations causing a decoder executed by the endpoint to generate a reconstructed video by at least providing the descriptor and the pivot image as an input to a generative model.

### Claim 11 (depends on 10)

The medium of claim 10, wherein the generative model generates intermediate frames of the reconstructed video between the pivot image and a second pivot image based at least in part on the descriptor.

### Claim 12 (depends on 9)

The medium of claim 9, wherein obtaining the pivot image further comprises sampling frames of the video over an interval of time.

### Claim 13 (depends on 9)

The medium of claim 9, wherein obtaining the pivot image further comprises causing a second machine learning model to determine the pivot image includes a conceptual element of the video.

### Claim 14 (depends on 9)

The medium of claim 9, wherein the medium further stores executable instructions, that, cause the processing device to perform operations causing the machine learning model to generate a second descriptor that includes a second natural language description of a relationship between the set of pivot images and at least one other pivot image obtained from the video, where the pivot image of the at least one other pivot image are provided to the machine learning model as an input.

### Claim 15 (depends on 9)

The medium of claim 9, wherein the machine learning model includes a large language model (LLM).

### Claim 16 (depends on 9)

The medium of claim 9, wherein obtaining the pivot image further comprises causing the machine learning model to generate a second natural language description of a frame of the video and selecting the frame as the pivot image based at least in part on the second natural language description.

### Claim 17

A method for video compression comprising:
obtaining a descriptor and a pivot image, the descriptor including a natural language description associated with the pivot image generated by a first machine learning model, the pivot image extracted from a video; and
causing a second machine learning model to generate reconstructed video based at least in part on the pivot image and the descriptor.

### Claim 18 (depends on 17)

The method of claim 17, wherein the descriptor further includes a second natural language description of objects within the pivot image.

### Claim 19 (depends on 17)

The method of claim 17, wherein causing the second machine learning model to generate the reconstructed video further comprises causing the second machine learning model to reconstruct a first version of the video.

### Claim 20 (depends on 17)

The method of claim 17, wherein causing the second machine learning model to generate the reconstructed video further comprises combining a plurality of images generated by the second machine learning model based at least in part on the pivot image and the descriptor.
