System and method for generating avatar of an active speaker in a meeting

One or more non-transitory computer readable storage medium storing instructions that when executed by one or more processors, cause:

• receiving at least one image of a face of a participant user;
• generating a static avatar data of the participant user based on the at least one image using a first machine learning (ML) model;
• receiving video data associated with the participant user, wherein the video data includes lips movement, eyebrows movement, jaw movement, and eyes movement of the participant user;
• using a second ML model to generate animated avatar data for generating an avatar video stream, wherein input for the second ML model includes the static avatar data, the video data, and a corresponding audio stream of the participant user, wherein the second ML model is trained using the participant user's own speech to mimic the participant user's specific lips movements and accent;
• wherein the animated avatar data mimics appropriate facial movements associated with the corresponding audio stream;
• determining, from a meeting audio stream associated with a meeting, that the participant user is an active speaker at the meeting;
• in response to determining that the participant user is the active speaker at the meeting, determining whether a meeting video stream associated with the meeting contains facial content of the active speaker;
• in response to determining that the participant user is the active speaker and that the meeting video stream does not include facial content of the active speaker, retrieving the animated avatar data associated with the active speaker;
• and
• generating, based on the animated avatar data, an avatar video stream of the active speaker mimicking the meeting audio stream, wherein the active speaker is the participant user.

A system, comprising:

• a processor;
• a memory operatively connected to the processor and storing instructions that, when executed by the processor, cause:
• receiving at least one image of a face of a participant user;
• generating a static avatar data of the participant user based on the at least one image using a first machine learning (ML) model;
• receiving video data associated with the participant user, wherein the video data includes lips movement, eyebrows movement, jaw movement, and eyes movement of the participant user;
• and
• using a second ML model to generate animated avatar data for generating an avatar video stream, wherein input for the second ML model includes the static avatar data, the video data, and a corresponding audio stream of the participant user, wherein the second ML model is trained using the participant user's own speech to mimic the participant user's specific lips movements and accent;
• wherein the animated avatar data mimics appropriate facial movements associated with the corresponding audio stream
• determining, from a meeting audio stream associated with a meeting, that the participant user is an active speaker at the meeting;
• in response to determining that the participant user is the active speaker at the meeting, determining whether a meeting video stream associated with the meeting contains facial content of the active speaker;
• in response to determining that the participant user is the active speaker and that the meeting video stream does not include facial content of the active speaker, retrieving the animated avatar data associated with the active speaker;
• and
• generating, based on the animated avatar data, an avatar video stream of the active speaker mimicking the meeting audio stream, wherein the active speaker is the participant user.