# US 2024/0176850 A1 · Systems and methods for editing audiovisual data using latent codes from generative networks and models

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2024/0176850 A1
- Document ID: US20240176850A1
- Status: Published application
- Kind: A1
- Application number: 17/093,380
- Filed: 2020-11-09
- Published: 2024-05-30
- Earliest claimed: 2019-11-08
- Assignees: Unknot Inc.
- Applicants: Unknot Inc.
- Inventors: Ross F. Elliot; Seth Haberman; Michael A. Baumer; Nakul Darwa
- Web page: https://patentgrep.com/patent/US20240176850A1

## CPC

- G06F 16/532: Query formulation, e.g. graphical querying
- G06F 16/535: Filtering based on additional data, e.g. user or group profiles
- G06F 18/2113: by ranking or filtering the set of features, e.g. using a measure of variance or of feature cross-correlation
- G06F 18/214: Generating training patterns; Bootstrap methods, e.g. bagging or boosting
- G06N 3/04: Architecture, e.g. interconnection topology
- G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU]
- G06N 3/045: Combinations of networks
- G06N 3/0455: Auto-encoder networks; Encoder-decoder networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/047: Probabilistic or stochastic networks
- G06N 3/0475: Generative networks
- G06N 3/0495: Quantised networks; Sparse networks; Compressed networks
- G06N 3/08: Learning methods
- G06N 3/09: Supervised learning
- G06N 3/094: Adversarial learning
- G06T 9/002: {using neural networks}
- G06V 10/82: using neural networks
- G06V 30/19173: {Classification techniques}
- G06V 30/194: References adjustable by an adaptive method, e.g. learning
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- H04N 19/90: using coding techniques not provided for in groups H04N19/10-H04N19/85, e.g. fractals
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

Systems and methods for viewing, storing, transmitting, searching, and editing application-specific audiovisual content (or other unstructured data) are disclosed in which edge devices generate content on the fly from a partial set of instructions rather than merely accessing the content in its final or near-final form. An image processing architecture may include a generative model that may be a deep learning model. The generative model may include a latent space comprising a plurality of latent codes and a trained generator mapping. The trained generator mapping may convert points in the latent space to uncompressed data points, which in the case of audiovisual content may be generated image frames. The generative model may be capable of closely approximating (up to noise or perceptual error) most or all potential data points in the relevant compression application, which in the case of audiovisual content may be source images.

## Claims

### Claim 1

A system for editing source images, the system comprising:
a representative data set including a plurality of source images;
a transformation module configured to implement intended transformations of source images and derived from the representative data set, the transformation module comprising:
a trained generative model comprising:
a latent space comprising a plurality of latent codes;
a trained generator mapping configured to map latent codes to generated image frames that share at least one characteristic feature with at least one of the source images in the representative data set;
a generator-coupled compressor mapping configured to map one or more of the source images to one or more resulting latent codes, wherein the trained generator mapping maps the one or more resulting latent codes to one or more reconstructed image frames resembling the one or more source images; and
one or more generator-coupled transformation identifiers, configured to transform one or more latent codes, the transformations corresponding to one or more intended transformations of one or more source images.

### Claim 2 (depends on 1)

The system of claim 1, further comprising a user input module configured to map the one or more intended transformations of source images to the one or more generator-coupled transformation identifiers that the transformation module uses to implement the intended transformations.

### Claim 3 (depends on 1)

The system of claim 1 wherein the trained generator mapping maps nearby latent codes to perceptually similar images

### Claim 4

A system for producing generator-coupled transformation identifiers corresponding to intended transformations, comprising:
a representative data set including a plurality of source images;
an untrained generative model comprising a latent space and an untrained generator mapping, the latent space comprising a plurality of latent codes, the untrained generator mapping using a plurality of parameters to map each latent code to a generated image;
a generator pretraining module configured to preprocess a first subset of the representative dataset;
a quality assessment module configured to convert image frames into quantities that assess the degree to which those image frames share characteristic features with image frames in the first subset of the representative data set preprocessed by the generator pretraining module;
a generator training module configured to modify the untrained generative model to optimize the result of the quality assessment module when applied to the output of the untrained generator mapping, thereby producing a trained generative model comprising the latent space and a trained generator mapping;
a transformation data set, comprising latent code and generated image frame pairs related by the trained generator mapping, each pair labeled by a plurality of transformation feature labels of the generated image frame;
an uncoupled transformation identifier using a plurality of parameters to transform one or more untransformed latent codes from the latent space of the trained generative model into transformed latent codes;
a transformation assessment module configured to assess the degree to which the uncoupled transformation identifier produces transformed latent codes, which, when input into the trained generator mapping, produce intended transformations with respect to the plurality of transformation feature labels; and
a transformation identifier training module configured to modify the uncoupled transformation identifier to optimize the result of the transformation assessment module when applied to the output of the uncoupled transformation identifier, thereby producing a generator-coupled transformation identifier.

### Claim 5

A method for editing source images, the method comprising:
defining a representative data set including a plurality of source images;
deriving a trained generative model from the representative data set, the trained generative model comprising a latent space including a plurality of latent codes;
mapping one or more of the source images to one or more resulting latent codes, wherein the mapping of the one or more resulting latent codes to one or more reconstructed image frames resembles the one or more source images;
transforming the one or more resulting latent codes using one or more generator-coupled transformation identifiers corresponding to one or more intended transformations; and
mapping the one or more resulting latent codes to generated image frames that share at least one characteristic feature with at least one of the source images in the representative data set.

### Claim 6 (depends on 1)

The method of claim 1, further comprising a user input module, configured to map the one or more intended transformations of the output of the trained generator mapping to the one or more generator-coupled transformation identifiers used by the transformation module to implement the intended transformations.

### Claim 7 (depends on 6)

The method of claim 6 further comprising maps nearby latent codes to perceptually similar images.
