System and method for voice driven lip syncing and head reenactment
AssigneeDe-Identification Ltd.
InventorsEliran Kuta, Sella Blondheim, Gil Perry, Amitay Nachmani, Matan Ben-Yosef, Or Gorodissky
Life of this patent
Estimating the term…
Abstract
A system and method voice driven animation of an object in an image by sampling an input video, depicting a puppet object, to obtain an image, receiving audio data, extracting voice related features from the audio data, producing an expression representation based on the voice related features, wherein the expression representation is related to a region of interest, obtaining from the image, auxiliary data related to the image and generating a target image based on the expression representation and the auxiliary data.
Claims18
A method for voice driven animation of an object in an image, the method comprising:
- sampling an input video, depicting a puppet object, to obtain an image;
- receiving audio data including a sound;
- extracting, by an expression representation (ER) model, voice related features from the audio data, wherein the voice related features provide information related to positions of face muscles while producing the sound;
- producing an appearance of the face while producing the sound based on the voice related features;
- obtaining, from the image, by a puppet data analyzer, auxiliary data related to content of the image;
- and
- generating, by an expression animation generator (EAG) model, a target image based on the appearance of the face while producing the sound and the auxiliary data,
- wherein the ER model, the puppet data analyzer and the EAG model are trained in a self-supervised manner by repeating the steps of sampling, extracting, producing, obtaining and generating a target image until a predetermined quality value related to a reconstruction quality of the target image is reached, wherein during the training stage the audio data is an audio signal of the input video.
A system for voice driven animation of an object in an image, the system comprising:
- a memory;
- and
- a processor configured to:
- sample an input video, depicting a puppet object, to obtain an image;
- receive audio data including a sound;
- extract, by an expression representation (ER) model, voice related features from the audio data, wherein the voice related features provide information related to positions of face muscles while producing the sound;
- produce an appearance of the face while producing the sound based on the voice related features;
- obtain, from the image, by a puppet data analyzer, auxiliary data related to content of the image;
- and
- generate, by an expression animation generator (EAG) model, a target image based on the appearance of the face while producing the sound and the auxiliary data,
- wherein the ER model, the puppet data analyzer and the EAG model are trained in a self-supervised manner by repeating the steps of sampling, extracting, producing, obtaining and generating a target image until a predetermined quality value related to a reconstruction quality of the target image is reached, wherein during the training stage the audio data is an audio signal of the input video.
A method for voice driven animation of an object in an image, the method comprising:
- analyzing, by an expression representation (ER) model, a received audio data including a sound to extract voice related features from the audio data, wherein the voice related features provide information related to positions of face muscles while producing the sound;
- generating, based on the voice related features, the appearance of the face while producing the sound;
- sampling an input video, depicting a puppet object, to obtain an image;
- predicting, from the image, by a puppet data analyzer, auxiliary data related to the puppet object and supplementary data related to a scene represented in the image;
- generating the facial area of the puppet object based on the appearance of the face while producing the sound and the auxiliary data;
- and
- generating, by an expression animation generator (EAG) model, a target image by combining the facial area of the puppet object and the supplementary data,
- wherein the ER model, the puppet data analyzer and the EAG model are trained in a self-supervised manner by repeating the steps of sampling, extracting, producing, obtaining and generating a target image until a predetermined quality value related to a reconstruction quality of the target image is reached, wherein during the training stage the audio data is an audio signal of the input video.
Dependencies are inferred from the claim text.