Real-time performance capture with on-the-fly correctives

A system, comprising:

• a memory storing a plurality of instructions;
• and
• one or more processors configurable to:
• obtain a three-dimensional scan of a subject;
• generate a customized digital model including a set of blendshapes using the three-dimensional scan, each of one or more blendshapes of the set of blendshapes representing at least a portion of a characteristic of the subject;
• receive input data of the subject, the input data including video data and depth data;
• track body deformations of the subject by fitting the input data using one or more of the blendshapes of the set;
• and
• fit a refined linear model onto the input data using one or more adaptive principal component analysis shapes.

A computer-readable memory storing a plurality of instructions executable by one or more processors, the plurality of instructions comprising:

• instructions that cause the one or more processors to obtain a three-dimensional scan of a subject;
• instructions that cause the one or more processors to generate a customized digital model including a set of blendshapes using the three-dimensional scan, each of one or more blendshapes of the set of blendshapes representing at least a portion of a characteristic of the subject;
• instructions that cause the one or more processors to receive input data of the subject, the input data including video data and depth data;
• instructions that cause the one or more processors to track body deformations of the subject by fitting the input data using one or more of the blendshapes of the set;
• and
• instructions that cause the one or more processors to fit a refined linear model onto the input data using one or more adaptive principal component analysis shapes.