Online modeling for real-time facial animation

A processing device, comprising:

• an input interface configured to receive tracking data corresponding to a facial expression of a user;
• a memory configured to store a dynamic expression model that includes a plurality of blendshapes and one or more corrective deformation fields;
• and
• a processing component coupled to the input interface and the memory, wherein the processing component is configured to:
• estimate tracking parameters based on the dynamic expression model and the tracking data, wherein the tracking parameters include a plurality of weights for the plurality of blendshapes and
• refine the dynamic expression model based on the tracking data and the estimated tracking parameters to produce a refined dynamic expression model, wherein the refining the dynamic model includes applying at least one of the one or more corrective deformation fields to each of the plurality of blendshapes.

A real-time facial animation system, comprising:

• a camera device, wherein the camera device is configured to generate tracking data corresponding to a facial expression of a user;
• and
• a processing device, wherein the processing device comprises:
• an input interface coupled to the camera device and configured to receive the tracking data,
• a memory configured to store a dynamic expression model, wherein the dynamic expression model includes a plurality of blendshapes and one or more corrective deformation fields for each of the blendshapes, and
• a processing component coupled to the input interface and the memory, wherein the processing component is configured to—
• estimate a corresponding plurality of weights for the plurality of blendshapes of the dynamic expression model based on the tracking data,
• generate a graphical representation corresponding to the facial expression of the user based on the plurality of weights, and
• refine the dynamic expression model based on the tracking data, the plurality of weights for the blendshapes, and the one or more corrective deformation fields.

A non-transitory program storage device comprising instructions stored thereon to cause one or more processors to:

• provide a dynamic expression model that includes a plurality of blendshapes and one or more corrective deformation fields;
• receive tracking data corresponding to a facial expression of a user;
• estimate tracking parameters based on the dynamic expression model and the tracking data, wherein the tracking parameters include a plurality of weights for the plurality of blendshapes;
• and
• refine the dynamic expression model based on the tracking data and the estimated tracking parameters to produce a refined dynamic expression model, wherein the refining includes applying at least one of the one or more corrective deformation fields to each of the plurality of blendshapes.