Systems and methods for providing a digital human in a virtual environment

A method, comprising:

• receiving, by a processor, from a user interacting with a system via a digital platform, a selection of a persona, for a digital human, from a plurality of personas;
• receiving, by the processor, in real-time, an input from the user via the digital platform;
• identifying, by the processor, a set of parameters associated with the input, wherein the set of parameters comprise at least a context of the input;
• determining, by the processor from a knowledge database, a response information based on the context of the input, wherein determining the response information from the knowledge database includes:
• generating embeddings associated with the input based on a multi-task model, wherein generating embeddings includes:
• processing and analyzing audio data of the input by performing speech to text conversion, tone detection, language detection, accent detection, and emotion detection, and
• converting the audio data into the generated embeddings based on the processing and anal y zing;
• selecting embeddings from the knowledge database based on the generated embeddings associated with the input;
• and
• determining a similarity parameter based on a comparison of the generated embeddings and the selected embeddings;
• generating, by the processor using a repository, a set of attributes for the response information based at least on the set of parameters associated with the input and the persona selected by the user, wherein the set of attributes correspond to at least an audio attribute and a visual attribute;
• aggregating, by the processor, the set of attributes and the response information to generate a personalized response to the input;
• and
• rendering, by the processor, the personalized response through the digital human on the digital platform,
• wherein the method further comprising:
• analyzing the audio data by:
• recording behavior of the user reading a pre-defined script in a controlled environment;
• extracting features from the audio data using a mel-frequency cepstral coefficients (MFCC) technique;
• encoding and decoding the extracted features using an encoder-decoder long short-term memory (LSTM), wherein an encoder of the LSTM reads the extracted features and outputs summarized information as internal state vectors, and a decoder of the LSTM generates an output sequence which is initialized from final states of the encoder;
• applying a connectionist temporal classification (CTC) to the output of the encoder to calculate a loss function associated with the audio data;
• performing a normalization on the output obtained from the encoder.

A non-transitory computer-readable medium comprising machine-readable instructions that are executable by a processor to:

• receive, from a user via a digital platform, a selection of a persona, for a digital human, from a plurality of personas;
• receive, in real-time, an input from the user via the digital platform;
• identify a set of parameters associated with the input, wherein the set of parameters comprise at least a context of the input;
• determine, from a knowledge database, a response information based on the context of the input, wherein determining the response information from the knowledge database includes:
• generating embeddings associated with the input based on a multi-task model, wherein generating embeddings includes:
• processing and analyzing audio data of the input by performing speech to text conversion, tone detection, language detection, accent detection, and emotion detection, and
• converting the audio data into the generated embeddings based on the processing and analyzing;
• selecting embeddings from the knowledge database based on the generated embeddings associated with the input;
• and
• determining a similarity parameter based on a comparison of the generated embeddings and the selected embeddings;
• generate, using a repository, a set of attributes for the response information based at least on the set of parameters associated with the input and the persona selected by the user, wherein the set of attributes correspond to at least an audio attribute and a visual attribute;
• aggregate the set of attributes and the response information to generate a personalized response to the input;
• and
• render the personalized response by the digital human on the digital platform,
• wherein the machine-readable instructions further cause the processor to:
• analyze the audio data by:
• recording behavior of the user reading a pre-defined script in a controlled environment;
• extracting features from the audio data using a mel-frequency cepstral coefficients (MFCC) technique;
• encoding and decoding the extracted features using an encoder-decoder long short-term memory (LSTM), wherein an encoder of the LSTM reads the extracted features and outputs summarized information as internal state vectors, and a decoder of the LSTM generates an output sequence which is initialized from final states of the encoder;
• applying a connectionist temporal classification (CTC) to the output of the encoder to calculate a loss function associated with the audio data;
• performing a normalization on the output obtained from the encoder.