Generating a realistic animated avatar of a user in real-time during a teleconference

A method for generating animated avatars of users in real-time during a teleconference, the method comprising:

• training a plurality of AI avatar generation models to create an avatar of a first user, wherein each AI avatar generation model is trained using different types of sensor data;
• deploying an AI avatar generation agent on a cloud server;
• in response to a teleconference call being initiated between a communication device of the first user and a communication device of the second user, collecting sensor data from a plurality of different sensors associated with the first user, wherein the sensor data comprises at least text from the first user;
• activating a data processing model and of the second user to identify the types of sensor data received from the communication device of the first user and activating the AI avatar generation agent to execute one or more of the plurality of AI avatar generation models, corresponding to the identified types of sensor data, for generating, based at least on the received sensor data, an animated avatar of the first user, wherein the animated avatar simulates at least one of:
• physical likeness, facial expressions, speech mannerisms, co-speech gestures, and voice of the first user during the teleconference,
• wherein the one or more of the plurality of AI avatar generation models comprises at least a voice generation model configured to generate computer-generated speech for the first user from text of the first user using the AI avatar generation agent on the cloud server;
• and
• displaying, on the communication device of second user, the animated avatar of the first user with the computer-generated speech for the first user synced with the text obtained from the first user in real-time during the teleconference.

A system for generating animated avatars of users in real-time during a teleconference, comprising:

• at least one memory;
• and
• at least one hardware processor coupled with the at least one memory and configured, individually or in combination, to:
• train a plurality of AI avatar generation models to create an avatar of a first user, wherein each AI avatar generation model is trained using different types of sensor data;
• deploy an AI avatar generation agent on a communication device of a second user or on a cloud server;
• in response to a teleconference call being initiated between a communication device of the first user and a communication device of the second user, collect sensor data from a plurality of different sensors associated with the first user, wherein the sensor data comprises at least real-time audio of voice of the first user;
• activate a data processing model to identify the types of sensor data received from the communication device of the first user and activating the AI avatar generation agent to execute one or more of the plurality of AI avatar generation models, corresponding to the identified types of sensor data, for generating, based at least on the received sensor data, an animated avatar of the first user, wherein the animated avatar simulates at least one of:
• physical likeness, facial expressions, speech mannerisms, co-speech gestures, and voice of the first user during the teleconference;
• and
• display, on the communication device of second user, the animated avatar of the first user synced with the real-time audio of the voice of the first user during the teleconference.