Avatar animation with general pretrained facial movement encoding

A method for training an expression encoder, comprising:

• obtaining a first frame and a second frame, the first frame and second frame including at least a portion of a face;
• generating a first expression feature for the first frame, the first expression feature representing a first expression of the face;
• generating a second expression feature for the second frame, the second expression feature representing a second expression of the face;
• generating a first view angle feature for the first frame, the first view angle feature representing a first angle from which the face is viewed from;
• generating a second view angle feature for the second frame, the second view angle feature representing a second angle from which the face is viewed from;
• crossing at least one of one of the first expression feature and second expression feature or the first view angle feature and the second view angle feature;
• determining a first loss value based on the crossing;
• and
• adjusting a feature encoder based on the determined first loss value.

An apparatus for generating a representation of a face, the apparatus comprising:

• at least one memory;
• and
• at least one processor coupled to the at least one memory, the at least one processor being configured to:
• obtain one or more images of a face;
• generate an encoded expression representing an expression of the face, wherein predetermined characteristics of the face remain constant relative to the encoded expression;
• map the encoded expression to a corresponding expression of a facial model;
• and
• generate the representation of the facial model based on the encoded expression.

An apparatus for training an expression encoder, the apparatus comprising:

• at least one memory;
• and
• at least one processor coupled to the at least one memory, the at least one processor being configured to:
• obtain a first frame and a second frame, the first frame and second frame including at least a portion of a face;
• generate a first expression feature for the first frame, the first expression feature representing a first expression of the face;
• generate a second expression feature for the second frame, the second expression feature representing a second expression of the face;
• generate a first view angle feature for the first frame, the first view angle feature representing a first angle from which the face is viewed from;
• generate a second view angle feature for the second frame, the second view angle feature representing a second angle from which the face is viewed from;
• cross at least one of one of the first expression feature and second expression feature or the first view angle feature and the second view angle feature;
• determine a first loss value based on the crossing;
• and
• adjust a feature encoder based on the determined first loss value.

A non-transitory computer-readable medium having stored thereon instructions that, when executed by at least one processor, cause the at least one processor to:

• obtain one or more images of a face;
• generate an encoded expression representing an expression of the face, wherein predetermined characteristics of the face remain constant relative to the encoded expression;
• map the encoded expression to a corresponding expression of a facial model;
• and
• generate a representation of the facial model based on the encoded expression.

A non-transitory computer-readable medium having stored thereon instructions that, when executed by at least one processor, cause the at least one processor to:

• obtain a first frame and a second frame, the first frame and second frame including at least a portion of a face;
• generate a first expression feature for the first frame, the first expression feature representing a first expression of the face;
• generate a second expression feature for the second frame, the second expression feature representing a second expression of the face;
• generate a first view angle feature for the first frame, the first view angle feature representing a first angle from which the face is viewed from;
• generate a second view angle feature for the second frame, the second view angle feature representing a second angle from which the face is viewed from;
• cross at least one of one of the first expression feature and second expression feature or the first view angle feature and the second view angle feature;
• determine a first loss value based on the crossing;
• and
• adjust a feature encoder based on the determined first loss value.