Video communication and streaming using diffusion guided by control data derived from audio

A computer-implemented method for generating image sequences, the method comprising:

• receiving, at a computing device, values of fine-tuning weights for a first artificial neural network implementing a pre-trained diffusion model and values of weights of a first control neural network wherein the weights of the first artificial network are generated by training the first artificial neural network using a first set of frames of a first set of training image data involving a subject and conditioning training data derived from the first set of frames of the first set of training image data and wherein the weights of the first control neural network are generated by training a first composite neural network including the first artificial neural network and the first control neural network, the training the first composite neural network including providing, to the first composite neural network, a second set of frames of a second set of training image data involving the subject and auxiliary training data derived from audio data associated with the second set of frames of the second set of training image data;
• wherein the first artificial neural network includes one or more layers with fixed weights implementing the pre-trained diffusion model and at least one trainable layer including the fine-tuning weights where the values of the fine-tuning weights are adjusted during the training of the first artificial neural network;
• configuring a second artificial neural network by inserting the values of the fine-tuning weights into an adaptable layer of the second artificial neural network wherein the second artificial neural network includes fixed-weight layers implementing the pre-trained diffusion model;
• modifying a second control neural network based upon the values of the weights of the first control neural network, the second control neural network and the second artificial neural network forming a second composite neural network;
• receiving, at the computing device, conditioning data derived from input frames of image data containing the subject and auxiliary data generated from audio data associated with the input frames of image data;
• generating, by the second composite neural network, images corresponding to the input frames of image data using the conditioning data and the auxiliary data.