Three-dimensional avatar generation system

A system comprising:

• a non-transitory storage medium storing computer program instructions;
• and
• a processor configured to execute the computer program instructions to cause operations comprising:
• receiving video data and non-video data from a user device for generation of an avatar corresponding to a user of the user device, the video data comprising a video of the user of the user device, the non-video data comprising information associated with a desired appearance and behavior of the avatar;
• generating the avatar reflective of the appearance and the behavior of the user by inputting the video data and non-video data to a generative machine learning model trained to generate highly realistic avatars for users, wherein the avatar looks, behaves, sounds, and interacts like the user, wherein the video data is pre-processed to remove background information from the video data and isolate the user within the video data;
• cloning a voice of the user for use with the avatar, wherein the cloning comprises extracting audio data from the video data and generating a synthetic voice for the avatar using a model trained on voice characteristics of the user, wherein speech output of the avatar matches one or more of a tone, pace, inflection, or accent of the user;
• storing the avatar and the cloned voice of the user in a network accessible location;
• and
• deploying the avatar in a third-party application for real-time or near real-time interaction with a second user, wherein responses of the avatar are generated using a large language model and are delivered with synchronized lip movements generated by a lip-sync module based on learned speech patterns of the user.

A non-transitory storage medium comprising computer program instructions stored thereon, which, when executed by a processor, causes a computing system to perform operations comprising:

• receiving, by the computing system, video data and non-video data from a user device for generation of an avatar corresponding to a user of the user device, the video data comprising a video of the user of the user device, the non-video data comprising information associated with a desired appearance and behavior of the avatar;
• generating, by the computing system, the avatar reflective of the appearance and the behavior of the user by inputting the video data and non-video data to a generative machine learning model trained to generate highly realistic avatars for users, wherein the avatar looks, behaves, sounds, and interacts like the user, wherein the video data is pre-processed to remove background information from the video data and isolate the user within the video data;
• cloning, by the computing system, a voice of the user for use with the avatar, wherein the cloning comprises extracting audio data from the video data and generating a synthetic voice for the avatar using a model trained on voice characteristics of the user, wherein speech output of the avatar matches one or more of a tone, pace, inflection, or accent of the user;
• storing, by the computing system, the avatar and the cloned voice of the user in a network accessible location;
• and
• deploying, by the computing system, the avatar in a third-party application for real-time or near real-time interaction with a second user, wherein responses of the avatar are generated using a large language model and are delivered with synchronized lip movements generated by a lip-sync module based on learned speech patterns of the user.