System and method for voice driven lip syncing and head reenactment
AssigneeDe-Identification Ltd.
InventorsEliran KUTA, Sella BLONDHEIM, Gil PERRY, Amitay NACHMANI, Matan BEN-YOSEF, Or GORODISSKY
Life of this patent
Estimating the term…
Abstract
A system and method voice driven animation of an object in an image by sampling an input video, depicting a puppet object, to obtain an image, receiving audio data, extracting voice related features from the audio data, producing an expression representation based on the voice related features, wherein the expression representation is related to a region of interest, obtaining from the image, auxiliary data related to the image and generating a target image based on the expression representation and the auxiliary data.
Claims20
A method for voice driven animation of an object in an image, the method comprising:
- sampling an input video, depicting a puppet object, to obtain an image;
- receiving audio data;
- extracting voice related features from the audio data;
- producing an expression representation based on the voice related features, wherein the expression representation is related to a region of interest;
- obtaining, from the image, auxiliary data related to the image;
- and
- generating a target image based on the expression representation and the auxiliary data.
A system for voice driven animation of an object in an image, the system comprising:
- a memory;
- and
- a processor configured to:
- sample an input video, depicting a puppet object, to obtain an image;
- receive audio data;
- extract voice related features from the audio data;
- produce an expression representation based on the voice related features, wherein the expression representation is related to a region of interest;
- obtain, from the image, auxiliary data related to the image;
- and
- generate a target image based on the expression representation and the auxiliary data.
A method for voice driven animation of an object in an image, the method comprising:
- analyzing a received audio data to extract voice related features from the audio data;
- generating, based on the voice related features, an expression representation;
- sampling an input video, depicting a puppet object, to obtain an image;
- predicting, from the image, auxiliary data related to the puppet object and supplementary data related to a scene represented in the image;
- generating a region of interest of the puppet object based on the expression representation and the auxiliary data;
- and
- generating a target image by combining the region of interest of the puppet object and the supplementary data.
Dependencies are inferred from the claim text.