System and method for generating a facial image from a voice sample using a stylegan
AssigneeCorsound AI Ltd
InventorAinur Mukhambetova
Life of this patent
Estimating the term…
Abstract
System and method for reconstructing a facial image of a speaker from a voice sample of the speaker may include providing the voice sample of the speaker to a trained voice encoder to generate a voice embedding of the speaker, wherein the voice encoder is trained to provide a voice embedding that matches an image embedding of the facial image of the speaker; providing the voice embedding of the speaker to a trained mapping network to generate an intermediate latent vector, wherein the mapping network is trained to generate an intermediate latent vector for a StyleGAN from the voice embedding; and providing the intermediate latent vector to the StyleGAN to generate the facial image of a speaker.
Claims19
A method for reconstructing a facial image of a speaker from a voice sample of the speaker, the method comprising:
- providing the voice sample of the speaker to a trained voice encoder to generate a voice embedding of the speaker, wherein the voice encoder is trained to provide a voice embedding that matches an image embedding of an input facial image of the speaker;
- providing the voice embedding of the speaker to a trained mapping network to generate an intermediate latent vector, wherein the mapping network is trained to generate an intermediate latent vector for a StyleGAN from the voice embedding;
- and
- providing the intermediate latent vector to the StyleGAN to generate the facial image of a speaker.
A method for generating a reconstructed facial image of a speaker from a voice sample of the speaker, the method comprising:
- in a training stage:
- obtaining a pretrained voice-face matching network comprising a voice encoder configured to generate a voice embedding from a voice sample and an image encoder configured to generate an image embedding from an input facial image, wherein the voice encoder and the image encoder are trained so that a distance between the voice embedding and the image embedding of a matching voice sample and facial image is less than the distance between the voice embedding and the image embedding of an unmatching voice and facial image;
- training a mapping network to generate an intermediate latent vector for a StyleGAN from the image embedding generated by the image encoder, so that the StyleGAN generates the reconstructed facial image;
- during inference:
- providing the voice sample of the speaker to the trained voice encoder to generate a voice embedding of the speaker;
- and
- generating an intermediate latent vector from the voice embedding of the speaker by providing the voice embedding of the speaker to the trained mapping network;
- and
- providing the intermediate latent vector generated from the voice embedding of the speaker to the StyleGAN, so that the StyleGAN generates the facial image of the speaker.
A system for reconstructing a facial image of a speaker from a voice sample of the speaker, the system comprising:
- a memory;
- and
- a processor configured to:
- provide the voice sample of the speaker to a trained voice encoder to generate a voice embedding of the speaker, wherein the voice encoder is trained to provide a voice embedding that matches an image embedding of an input facial image of the speaker;
- provide the voice embedding of the speaker to a trained mapping network to generate an intermediate latent vector, wherein the mapping network is trained to generate an intermediate latent vector for a StyleGAN from the voice embedding;
- and
- provide the intermediate latent vector to the StyleGAN to generate the facial image of a speaker.
Dependencies are inferred from the claim text.