Back to results
US 12,591,772 B1Granted
Techniques for training identity transfer models
AssigneeAmazon Technologies, Inc.
InventorsWenbin Ouyang, Naveen Sudhakaran Nair
Life of this patent
Estimating the term…
Abstract
Systems, devices, and methods are provided for training and/or inferencing using machine-learning models. In at least one embodiment, an identity embedding and a content embedding are extracted from training media data, such as training audio data and/or training visual data. The identity embedding may encode information relating to a person's voice and/or visual properties. A decoder may be trained to create synthetic media, such as audio and visual content, based on identity and content embeddings.
Claims20
A computer-implemented method, comprising:
- training a first autoencoder, comprising a first encoder and a first decoder, the first autoencoder trained to at least:
- obtaining first training audio data from a plurality of audio training data;
- obtaining a first training spectrogram of the first training audio data;
- splitting the first training spectrogram into a plurality of training segments;
- selecting a first training segment of the plurality of training segments;
- providing the first training segment as a training input to the first encoder to determine a first training voice identity embedding;
- providing the first training voice identity embedding to the first decoder to determine a first training output;
- computing a first loss based on a first difference between the first training output and a second training segment of the plurality of training segments;
- and
- updating a first set of parameters of the first autoencoder by at least minimizing the first loss;
- training a second autoencoder comprising a second encoder and a second decoder, the second autoencoder trained to at least:
- applying a transformation to the first training audio data to produce a first transformed training audio data;
- obtaining a first transformed training spectrogram of the first transformed training audio data;
- providing the first transformed training spectrogram to the second encoder to determine a first training content embedding;
- obtaining a first voice identity embedding from a trained first autoencoder;
- providing the first training content embedding and the first voice identity embedding to the second decoder to determine a second training output;
- determining training ground truth for the second training output;
- computing a second loss based on a second difference between the second training output and the training ground truth and a similarity between the content embedding and the training ground truth;
- and
- updating a second set of parameters of the second autoencoder by at least minimizing the second loss;
- and
- providing the first set of parameters of the first autoencoder and the second set of parameters of the second autoencoder for inferencing.
19 more elements
A system, comprising:
- one or more processors;
- and
- memory storing executable instructions that, as a result of execution by the one or more processors, cause the system to:
- train a first autoencoder, comprising a first encoder and a first decoder, the first autoencoder trained to at least:
- obtaining first training audio data from a plurality of audio training data associated with a first identity;
- obtaining a first training spectrogram of the first training audio data;
- splitting the first training spectrogram into a plurality of training segments;
- selecting a first training segment of the plurality of training segments;
- providing the first training segment as a training input to the first encoder to determine a first training identity embedding;
- providing the first training identity embedding to the first decoder to determine a first training output;
- computing a first loss based on a first difference between the first training output and a second training segment of the plurality of training segments;
- and
- updating a first set of parameters of the first autoencoder by at least minimizing the first loss;
- train a second autoencoder comprising a second encoder and a second decoder, the second autoencoder trained to at least:
- applying a transformation to the first training audio data to produce a first transformed training audio data;
- obtaining a first transformed training spectrogram of the first transformed training audio data;
- providing the first transformed training spectrogram to the second encoder to determine a first training content embedding;
- obtaining a first identity embedding from a trained first autoencoder;
- providing the first training content embedding and the first identity embedding to the second decoder to determine a second training output;
- determine training ground truth for the second training output;
- computing a second loss based on a second difference between the second training output and the training ground truth and a similarity between the content embedding and the first identity embedding;
- and
- updating a second set of parameters of the second autoencoder by at least minimizing the second loss;
- and
- provide at least the second set of parameters of the second autoencoder for inferencing.
22 more elements
A non-transitory computer-readable storage medium storing executable instructions that, as a result of being executed by one or more processors of a computer system, cause the computer system to at least:
- train a first autoencoder, comprising a first encoder and a first decoder, the first autoencoder trained to at least:
- obtaining first training audio data from a plurality of audio training data associated with a first identity;
- obtaining a first training spectrogram of the first training audio data;
- splitting the first training spectrogram into a plurality of training segments;
- selecting a first training segment of the plurality of training segments;
- providing the first training segment as a training input to the first encoder to determine a first training identity embedding;
- providing the first training identity embedding to the first decoder to determine a first training output;
- computing a first loss based on a first difference between the first training output and a second training segment of the plurality of training segments;
- and
- updating a first set of parameters of the first autoencoder by at least minimizing the first loss;
- train a second autoencoder comprising a second encoder and a second decoder, the second autoencoder trained to at least:
- applying a transformation to the first training audio data to produce a first transformed training audio data;
- obtaining a first transformed training spectrogram of the first transformed training audio data;
- providing the first transformed training spectrogram to the second encoder to determine a first training content embedding;
- obtaining a first identity embedding from a trained first autoencoder;
- providing the first training content embedding and the first identity embedding to the second decoder to determine a second training output;
- determine training ground truth for the second training output;
- computing a second loss based on a second difference between the second training output and the training ground truth and a similarity between the content embedding and the first identity embedding;
- and
- updating a second set of parameters of the second autoencoder by at least minimizing the second loss;
- and
- provide at least the second set of parameters of the second autoencoder for inferencing.
19 more elements
Dependencies are inferred from the claim text.