Method and system for implementing three-dimensional facial modeling and visual speech synthesis

A device, comprising:

• a display;
• one or more processors in communication with an image sensor, an accelerometer, and the display;
• and
• a non-transitory computer readable medium in communication with the one or more processors, the non-transitory computer readable medium having encoded thereon a set of instructions executable by the one or more processors to cause the device to:
• receive an input image comprising a face;
• determine an orientation, a size, and a location of the face in the input image;
• retrieve a three-dimensional model template comprising a face and a head;
• project the input image onto the three-dimensional model template comprising the face and the head to generate a three-dimensional model corresponding to the input image;
• define, on the three-dimensional model, a polygon mesh in a region of at least one facial feature;
• adjust parameters on the three-dimensional model, the region of the at least one facial feature corresponding to at least one facial feature in the input image, wherein scaling the three-dimensional model comprises:
• defining a first box to frame the face in the input image;
• defining a second box to frame the face of the three-dimensional model template;
• and
• scaling the second box of the three-dimensional model template by performing at least one of scaling to fit the first box of the input image or scaling so that the second box is centered on the first box in the input image;
• and
• display the three-dimensional model with the face of the input image projected onto the three-dimensional model.

An apparatus, comprising:

• one or more processors;
• and
• a non-transitory computer readable medium having encoded thereon a set of instructions executable by the one or more processors to cause the apparatus to:
• receive an input image comprising a face;
• determine an orientation, a size, and a location of the face in the input image;
• retrieve a three-dimensional model template comprising a face and a head;
• project the input image onto the three-dimensional model template comprising the face and the head to generate a three-dimensional model corresponding to the input image;
• define, on the three-dimensional model, a polygon mesh in a region of at least one facial feature;
• adjust parameters on the three-dimensional model, the region of the at least one facial feature corresponding to at least one facial feature in the input image, wherein scaling the three-dimensional model comprises:
• defining a first box to frame the face in the input image;
• defining a second box to frame the face of the three-dimensional model template;
• and
• scaling the second box of the three-dimensional model template by performing at least one of scaling to fit the first box of the input image or scaling so that the second box is centered on the first box in the input image;
• and
• display the three-dimensional model with the face of the input image projected onto the three-dimensional model.

A method, comprising:

• parsing, with a computing system, a string of text into one or more allophonic units of a plurality of allophonic units;
• encoding, with the computing system, each of the one or more allophonic units into a point or a trajectory in linguistic space, the linguistic space corresponding to a range of mouth movements in at least three dimensions, and the plurality of allophonic units collectively tracing a set of continuous trajectories in the linguistic space;
• retrieving, with the computing system and from a codebook that is stored in a data store, a plurality of indexed images or morphs, each of the indexed images or morphs corresponding to a respective encoded point in the linguistic space, the codebook comprising an index of images or morphs;
• rendering, with the computing system, the plurality of indexed images or morphs into an animation of the three-dimensional model;
• synchronizing, with the computing system, the animation of the three-dimensional model with an audio representation of the string of text;
• and
• outputting, with the computing system, the synchronized animation and the audio representation.

A device, comprising:

• one or more processors;
• and
• a non-transitory computer readable medium in communication with the one or more processors, the non-transitory computer readable medium having encoded thereon a set of instructions executable by the one or more processors to cause the device to:
• parse a string of text into one or more allophonic units of a plurality of allophonic units;
• encode each of the one or more allophonic units into a point or a trajectory in linguistic space, the linguistic space corresponding to a range of mouth movements in at least three dimensions, and the plurality of allophonic units collectively tracing a set of continuous trajectories in the linguistic space;
• retrieve, from a codebook that is stored in a data store, a plurality of indexed images or morphs, each of the indexed images or morphs corresponding to a respective encoded point in the linguistic space, the codebook comprising an index of images or morphs;
• render the plurality of indexed images or morphs into an animation of the three-dimensional model;
• synchronize the animation of the three-dimensional model with an audio representation of the string of text;
• and
• output the synchronized animation and the audio representation.

An apparatus, comprising:

• one or more processors;
• and
• a non-transitory computer readable medium having encoded thereon a set of instructions executable by the one or more processors to cause the apparatus to:
• parse a string of text into one or more allophonic units of a plurality of allophonic units;
• encode each of the one or more allophonic units into a point or a trajectory in linguistic space, the linguistic space corresponding to a range of mouth movements in at least three dimensions, and the plurality of allophonic units collectively tracing a set of continuous trajectories in the linguistic space;
• retrieve, from a codebook that is stored in a data store, a plurality of indexed images or morphs, each of the indexed images or morphs corresponding to a respective encoded point in the linguistic space, the codebook comprising an index of images or morphs;
• render the plurality of indexed images or morphs into an animation of the three-dimensional model;
• synchronize the animation of the three-dimensional model with an audio representation of the string of text;
• and
• output the synchronized animation and the audio representation.

A method, comprising:

• receiving, with a computing system, an input image comprising a face;
• determining, with the computing system, an orientation, a size, and a location of the face in the input image;
• retrieving, with the computing system, a three-dimensional model template comprising a face and a head;
• projecting, with the computing system, the input image onto the three-dimensional model template comprising the face and the head to generate a three-dimensional model corresponding to the input image;
• defining, with the computing system and on the three-dimensional model, a polygon mesh in a region of at least one facial feature;
• adjusting, with the computing system, parameters on the three-dimensional model, the region of the at least one facial feature corresponding to at least one facial feature in the input image;
• displaying, with the computing system, the three-dimensional model with the face of the input image projected onto the three-dimensional model;
• parsing, with the computing system, a string of text into one or more allophonic units of a plurality of allophonic units;
• encoding, with the computing system, each of the one or more allophonic units into a point or a trajectory in linguistic space, the linguistic space corresponding to a range of mouth movements in at least three dimensions, and the plurality of allophonic units collectively tracing a set of continuous trajectories in the linguistic space;
• retrieving, with the computing system and from a codebook that is stored in a data store, a plurality of indexed images or morphs, each of the indexed images or morphs corresponding to a respective encoded point in the linguistic space, the codebook comprising an index of images or morphs;
• rendering, with the computing system, the plurality of indexed images or morphs into an animation of the three-dimensional model;
• synchronizing, with the computing system, the animation of the three-dimensional model with an audio representation of the string of text;
• and
• outputting, with the computing system, the synchronized animation and the audio representation.