Puppeteering remote avatar by facial expressions

A system comprising:

• data processing hardware;
• and
• memory hardware in communication with the data processing hardware, the memory hardware storing instructions that when executed on the data processing hardware cause the data processing hardware to perform operations comprising:
• receiving, from a first user device associated with a first user, a first captured image comprising a first facial framework of a face of the first user and associated audio data spoken by the first user;
• identifying, from the first facial framework, a facial cavity corresponding to at least one of an eye of the first user or a mouth of the first user;
• rendering the facial cavity onto the first facial framework;
• determining a facial texture corresponding to the face of the first user based on the first facial framework with the rendered facial cavity;
• transmitting, to a second user device associated with a second user, the facial texture as a three-dimensional avatar corresponding to a virtual representation of the face of the first user and the associated audio data spoken by the first user, the second user device configured to synchronously display the three-dimensional avatar and output the associated audio data;
• receiving, from the first user device, a second captured image comprising a second facial framework of the face of the first user;
• updating the facial texture based on the second facial framework;
• and
• transmitting, to the second user device, the updated facial texture as the three-dimensional avatar corresponding to the virtual representation of the face of the first user.