Puppeteering remote avatar by facial expressions

A system comprising:

• data processing hardware;
• and
• memory hardware in communication with the data processing hardware, the memory hardware storing instructions that when executed on the data processing hardware cause the data processing hardware to perform operations comprising:
• receiving a first facial framework and a first captured image of a face of a user with a neutral facial expression, the first facial framework corresponding to the face of the user at a first frame and comprising a first facial mesh of facial information;
• projecting the first captured image of the face onto the first facial framework;
• determining a facial texture corresponding to the face of the user based on the projected first captured image;
• receiving a second facial framework, the second facial framework corresponding to the face of the user at a second frame and comprising a second facial mesh of facial information;
• updating the facial texture based on the received second facial framework;
• displaying the updated facial texture as a three-dimensional avatar, the three-dimensional avatar corresponding to a virtual representation of the face of the user;
• and
• generating a rendition of an eye or a mouth of the user by:
• detecting edges of the eye or the mouth;
• determining that a sum of angles associated with the edges of the eye or the mouth correspond to a value of about two pi;
• approximating a position of the eye or the mouth based on the detected edges that correspond to the value of about two pi;
• extracting the mouth or the eye at the approximated position from the captured image of the face;
• and
• rendering the extracted mouth or the extracted eye at the approximated position with a fill.

A method comprising:

• receiving, at data processing hardware, a first facial framework and a first captured image of a face of a user with a neutral facial expression, the first facial framework corresponding to the face of the user at a first frame and comprising a first facial mesh of facial information;
• projecting, by the data processing hardware, the first captured image of the face onto the first facial framework;
• determining, by the data processing hardware, a facial texture corresponding to the face of the user based on the projected first captured image;
• and
• displaying, by the data processing hardware, the determined facial texture as a three-dimensional avatar, the three-dimensional avatar corresponding to a virtual representation of the face of the user;
• and
• generating a rendition of an eye or a mouth of the user by:
• detecting edges of the eye or the mouth;
• determining that a sum of angles associated with the edges of the eye or the mouth correspond to a value of about two pi;
• approximating a position of the eye or the mouth based on the detected edges that correspond to a value of about two pi;
• extracting the mouth or the eye at the approximated position from the captured image of the face;
• and
• rendering the extracted mouth or the extracted eye at the approximated position with a fill.