Audio-visual selection process for the synthesis of photo-realistic talking-head animations

A method for the synthesis of photo-realistic animation of an object, the method utilizing a first database of image samples showing an object in a plurality of appearances, a second database of visual features for each image sample of the object, and a third database of non-visual features of the object, the method comprising:

• obtaining, for each frame in a plurality of N frames of an object animation, a target feature vector comprising visual features and non-visual features associated with the object animation;
• selecting, via a processor and an audio-video unit selection process, longest possible candidate image samples from the first database utilizing the target feature vector to generate a photo-realistic animation of the object;
• calculating, for each pair of candidates of two consecutive frames, a concatenation cost from a combination of the visual features from the second database and the non-visual features from the third database;
• and
• performing a Viterbi search to find a least expensive path through the each pair of candidates accumulating a target cost and the concatenation cost.

A non-transitory computer-readable media storing an animation of an object generated according to a process of:

• creating a first database of image samples showing an object in a plurality of appearances;
• creating a second database of visual features for each image sample of the object;
• creating a third database of non-visual features of the object in each image sample;
• obtaining, for each frame in a plurality of N frames of an object animation, a target feature vector comprising visual features and non-visual features associated with the object animation;
• selecting, via a processor and an audio/video unit selection process, longest possible candidate image samples from the first database utilizing the target feature vector to generate a photo-realistic animation of the object;
• calculating, for each pair of candidates of two consecutive frames, a concatenation cost from a combination of the visual features from the second database and the non-visual features from the third database;
• and
• performing a Viterbi search to find the least expensive path through the candidates accumulating a target cost and the concatenation cost.

A system for synthesizing a photo-realistic animation of an object, the system comprising:

• a processor;
• a first module controlling the processor to create a first database of image samples showing the object in a plurality of appearances;
• a second module controlling the processor to create a second database of visual features for each image sample of the object;
• a third module controlling the processor to create a third database of non-visual features of the object;
• a fourth module controlling the processor to obtain, for each frame in a plurality of N frames of an object animation, a target feature vector comprising visual features and non-visual features associated with the object animation;
• a fifth module controlling the processor to select, via a processor and an audio/video unit selection process, longest possible candidate image samples from the first database utilizing the target feature vector to generate the photo-realistic animation of the object;
• a sixth module controlling the processor to calculate, for each pair of candidates of two consecutive frames, a concatenation cost from a combination of the visual features from the second database and the non-visual features from the third database;
• and
• a seventh module controlling the processor to perform a Viterbi search to find a least expensive path through the candidates accumulating a target cost and the concatenation cost.

A non-transitory computer-readable storage medium storing instructions which, when executed by a computing device, cause the computing device to synthesize photo-realistic animation of an object, the instructions utilizing a first database of image samples showing an object in a plurality of appearances, a second database of visual features for each image sample of the object, and a third database of non-visual features of the object, the instructions comprising:

• obtaining, for each frame in a plurality of N frames of an object animation, a target feature vector comprising visual features and non-visual features associated with the object animation;
• selecting, via a processor and an audio/video unit selection process, longest possible candidate image samples from the first database utilizing the target feature vector to generate a photo-realistic animation of the object;
• calculating, for each pair of candidates of two consecutive frames, a concatenation cost from a combination of the visual features from the second database and the non-visual features from the third database;
• and
• performing a Viterbi search to find the least expensive path through the candidates accumulating a target cost and the concatenation cost.

A system for animating an object, the system utilizing a first database of image samples showing an object in a plurality of appearances, a second database of visual features for each image sample of the object, and a third database of non-visual features of the object, system comprising:

• a processor;
• and
• a first module controlling the processor to animate an object generated according to steps comprising:
• obtaining, for each frame in a plurality of N frames of an object animation, a target feature vector comprising visual features and non-visual features associated with the object animation;
• selecting, via the processor and using an audio/video unit selection process, longest possible candidate image samples from the first database utilizing the target feature vector to generate a photo-realistic animation of the object;
• calculating, for each pair of candidates of two consecutive frames, a concatenation cost from a combination of the visual features from the second database and the non-visual features from the third database;
• and
• performing a Viterbi search to find the least expensive path through the candidates accumulating a target cost and the concatenation cost.