Sentiment-based adaptation of digital human responses

An apparatus comprising:

• at least one processing device comprising a processor coupled to a memory;
• the at least one processing device being configured to implement the following steps:
• determining, using at least one processing device, a sentiment of at least one user, wherein the sentiment of the at least one user comprises two or more of (i) a vocal sentiment, determined using at least one vocal sentiment analysis model that analyzes at least one audio stream associated with the at least one user, (ii) a text sentiment, determined using at least one text sentiment analysis model that analyzes at least one text stream, derived from the at least one audio stream, associated with the at least one user, and (iii) a facial sentiment, determined using at least one facial sentiment analysis model that analyzes one or more objects derived from one or more images of at least one video visual stream associated with the at least one user;
• applying the determined sentiment of the at least one user, as one or more sentiment-based prompts, to at least one artificial intelligence (AI)-based language model that determines at least one sentiment-tagged response to an input of the at least one user based at least in part on the determined sentiment of the at least one user, wherein the at least one sentiment-tagged response comprises at least one predicted sentiment label identifying at least one sentiment to be employed by at least one AI-based digital human when delivering the at least one sentiment-tagged response to the at least one user, wherein the at least one AI-based language model is trained to perform sentiment insertion into the at least one sentiment-tagged response;
• and
• providing the sentiment-tagged response to the at least one AI-based digital human for delivery to the at least one user, wherein the at least one AI-based digital human transforms at least a portion of the sentiment-tagged response into a spoken format using the at least one predicted sentiment label and at least one text-to-speech model.

A non-transitory processor-readable storage medium having stored therein program code of one or more software programs, wherein the program code when executed by at least one processing device causes the at least one processing device to perform the following steps:

• determining, using at least one processing device, a sentiment of at least one user, wherein the sentiment of the at least one user comprises two or more of (i) a vocal sentiment, determined using at least one vocal sentiment analysis model that analyzes at least one audio stream associated with the at least one user, (ii) a text sentiment, determined using at least one text sentiment analysis model that analyzes at least one text stream, derived from the at least one audio stream, associated with the at least one user, and (iii) a facial sentiment, determined using at least one facial sentiment analysis model that analyzes one or more objects derived from one or more images of at least one visual stream associated with the at least one user;
• applying the determined sentiment of the at least one user, as one or more sentiment-based prompts, to at least one artificial intelligence (AI)-based language model that determines at least one sentiment-tagged response to an input of the at least one user based at least in part on the determined sentiment of the at least one user, wherein the at least one sentiment-tagged response comprises at least one predicted sentiment label identifying at least one sentiment to be employed by at least one AI-based digital human when delivering the at least one sentiment-tagged response to the at least one user, wherein the at least one AI-based language model is trained to perform sentiment insertion into the at least one sentiment-tagged response;
• and
• providing the sentiment-tagged response to the at least one AI-based digital human for delivery to the at least one user, wherein the at least one AI-based digital human transforms at least a portion of the sentiment-tagged response into a spoken format using the at least one predicted sentiment label and at least one text-to-speech model.