# US 2025/0166290 A1 · Video communication and streaming using diffusion guided by control data derived from audio

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2025/0166290 A1
- Document ID: US20250166290A1
- Status: Published application
- Kind: A1
- Application number: 18/952,579
- Filed: 2024-11-19
- Published: 2025-05-22
- Earliest claimed: 2023-11-20
- Assignees: IKIN, Inc.
- Applicants: IKIN, Inc.
- Inventors: See XIONG; James Charles STIEFELMAIER; Bryan Lloyd WESTCOTT
- Web page: https://patentgrep.com/patent/US20250166290A1

## CPC

- G06T 15/20: Perspective computation
- G06F 40/58: Use of machine translation, e.g. for multi-lingual retrieval, for server-side translation for client devices or for real-time translation
- G06N 3/045: Combinations of networks
- G06N 3/08: Learning methods
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06N 3/096: Transfer learning
- G06T 5/70: Denoising; Smoothing
- G06T 2200/04: involving 3D image data
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/30201: Face
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

A computer-implemented method for image generation includes receiving conditioning data derived from imagery within input data frames. Auxiliary audio data derived from words or sounds vocalized by a subject within the input data frames is received. The audio data is included within audio content associated the input data frames. The conditioning data and the auxiliary audio data is provided to a composite artificial neural network where the composite artificial neural network is configured to perform a controlled diffusion process. The composite artificial neural network includes a neural network implementing a diffusion model in combination with a control neural network. The composite artificial neural network generates reconstructed versions of the input data frames based upon the conditioning data and the auxiliary audio data.

## Claims

### Claim 1

A computer-implemented method for generating image sequences, the method comprising:
receiving, at a computing device, values of fine-turning weights for a first artificial neural network implementing a pre-trained diffusion model and values of weights of a first control neural network wherein the weights of the first artificial network are generated by training the first artificial neural network using a first set of frames of training image data involving a subject and conditioning training data derived from the first set of frames of training image data and wherein the weights of the first control network are generated by training a first composite neural network including the first artificial neural network and the first control neural network, the training the first composite neural network including providing, to the first composite neural network, a second set of frames of training image data involving the subject and auxiliary training data derived from audio data associated with the second set of frames of training image data;
wherein the first artificial neural network includes one or more layers with fixed weights implementing the pre-trained diffusion model and at least one trainable layer including the fine-tuning weights where the values of the fine-tuning weights are adjusted during the training of the first artificial neural network;
configuring a second artificial neural network by inserting the values of the fine-tuning weights into an adaptable layer of the second artificial neural network wherein the second artificial neural network includes fixed-weight layers implementing the pre-trained diffusion model;
modifying a second control neural network based upon the values of the weights of the first control neural network, the second control neural network and the second artificial neural network forming a second composite neural network;
receiving, at the computing device, conditioning data derived from input frames of image data containing the subject and auxiliary data generated from audio data associated with the input frames of image data;
generating, by the second composite neural network, images corresponding to the input frames of image data using the conditioning data and the auxiliary data.

### Claim 2 (depends on 1)

The method of claim 1 wherein the first control neural network includes a trainable copy of one or more layers of the first artificial neural network, the values of the weights of the first control network are adjusted during the training of the first composite neural network, and the values of the weights of the first artificial neural network are not adjusted during the training of the first composite neural network.

### Claim 3 (depends on 1)

The method of claim 1 wherein the auxiliary data includes one of phonetic data and sentiment data associated with the subject.

### Claim 4 (depends on 1)

The method of claim 1 wherein the values of the set of fine-tuning weights correspond to low-rank adaptation (LoRA) parameter values.

### Claim 5 (depends on 1)

The method of claim 1 wherein the auxiliary data is time aligned with the conditioning data.

### Claim 6 (depends on 1)

The method of claim 1 wherein the audio data includes speech of the subject in a first language and wherein the auxiliary data is generated in part by translating the speech of the subject to a second language.

### Claim 7 (depends on 1)

The method of claim 1 wherein the conditioning data corresponds to one of compressed representations of the input frames of image data and sparse representations of the input frames of image data.

### Claim 8 (depends on 1)

The method of claim 1 wherein the first set of frames of training image data include a face of the subject.

### Claim 9 (depends on 8)

The method of claim 8 wherein the conditioning training data includes a first set of three-dimensional coordinate locations corresponding to facial landmarks of the face.

### Claim 10 (depends on 1)

The method of claim 1 wherein the generating includes:
sampling from a normal distribution to generate a set of sampled values;
performing denoising operations in accordance with the diffusion model.

### Claim 11

A computer-implemented method, comprising:
generating a set of fine-tuning weights for a pre-trained diffusion model, the generating including training a first artificial neural network using a first set of frames of training image data involving a subject and conditioning training data derived from the first set of frames of training image data, the first artificial neural network having one or more layers with fixed weights implementing the pre-trained diffusion model and at least one trainable layer including the fine-tuning weights where values of the fine-tuning weights are adjusted during the training;
training a first composite neural network including the first artificial neural network and a first control neural network, the training including providing, to the first composite neural network, a second set of frames of training image data involving the subject and auxiliary training data derived from audio data associated with the second set of frames of training image data;
sending, to a computing device, the values of the fine-tuning weights and values of weights of the first control neural network wherein the computing device is disposed to (i) insert the values of the fine-tuning weights into an adaptable layer of a second artificial neural network having fixed-weight layers implementing the pre-trained diffusion mode and (ii) modify a second control neural network based upon the values of the weights of the first control neural network, the second control neural network and the second artificial neural network forming a second composite neural network;
deriving conditioning data from input frames of image data containing the subject;
generating auxiliary data from audio data associated with the input frames of image data; and
sending the conditioning data and the auxiliary data to the computing device wherein the second composite neural network is configured to generate images corresponding to the input frames of image data using the conditioning data and the auxiliary data.

### Claim 12 (depends on 11)

The method of claim 11 wherein the first control neural network includes a trainable copy of one or more layers of the first artificial neural network, the method further including adjusting values of the weights of the first control network during the training of the first composite neural network wherein the values of the weights of the first artificial neural network are not adjusted during the training of the first composite neural network.

### Claim 13 (depends on 11)

The method of claim 11 wherein the auxiliary data includes one of phonetic data and sentiment data associated with the subject.

### Claim 14 (depends on 11)

The method of claim 11 further including time aligning the auxiliary data with the conditioning data.

### Claim 15 (depends on 11)

The method of claim 11 wherein the audio data includes speech of the subject in a first language, the method further including generating the auxiliary data in part by translating the speech of the subject to a second language.

### Claim 16 (depends on 11)

The method of claim 11 wherein the values of the set of fine-tuning weights correspond to low-rank adaptation (LoRA) parameter values.

### Claim 17 (depends on 11)

The method of claim 11 wherein the conditioning data corresponds to one of compressed representations of the input frames of image data and sparse representations of the input frames of image data.

### Claim 18 (depends on 11)

The method of claim 11 wherein the first set of frames of training image data include a face of the subject.

### Claim 19 (depends on 18)

The method of claim 18 wherein the conditioning training data includes a first set of three-dimensional coordinate locations corresponding to facial landmarks of the face.

### Claim 20 (depends on 19)

The method of claim 19 wherein the conditioning data includes a second set of three-dimensional coordinate locations corresponding to the facial landmarks of the face wherein the second set of coordinate locations are different from the first set of three-dimensional coordinate locations.
