Real-time system for spoken natural stylistic conversations with large language models

A system comprising:

• a processing unit;
• and
• a computer readable medium having encoded thereon computer-readable instructions that when executed by the processing unit cause the system to:
• configure a large language model with a conversational profile comprising training data that includes a set of example conversations that comprise labeled data demonstrating positive interactions, negative interactions, and appropriate responses and inappropriate responses, and a set of sentiments;
• receive a user input comprising a speech audio input;
• convert the user input into a text translation of the speech audio input;
• analyze the speech audio input to determine at least one of a volume, a tone of voice, and one or more inflections of a voice included in the speech audio input;
• analyze at least one of the volume, the tone of voice, and the one or more inflections with the text translation of the speech audio input using a prompt engine to determine a sentiment of the user input;
• generate a prompt, use the prompt engine, wherein the prompt includes the sentiment of the user input that is determined using at least one of the tone, the volume, the one or more inflections, and the text translation of the speech audio input, the prompt further comprising instructions for causing the large language model to include punctuation and a word selection in a text response that represents a selected sentiment from the set of sentiments within the conversational profile;
• cause the large language model to generate the text response based on the prompt and the text translation of the speech audio input, using the selected sentiment and the training data, wherein the training data causes the large language model to set patterns for subsequent text responses and prompts by increasing emphasis for positive user interactions and appropriate text responses and decreasing emphasis for negative user interactions and inappropriate text responses;
• receive the text response with the selected sentiment from the large language model generated from the prompt;
• select a style cue for an audio output based on the sentiment of the user input, the selected sentiment of the text response, and the prompt using the large language model;
• and
• generate the audio output using the text response and the style cue, wherein the audio output response uses the style cue, the punctuation, and the world selection from the text response to generate the audio output including at least one of an audio output volume, an audio output tone of voice, or one or more inflections of a voice in the audio output, wherein at least one of the audio output volume, the audio output tone of voice, or one or more inflections of the voice in the audio output corresponds to the style cue, the punctuation, and the word selection of the text response.

A storage device having encoded thereon computer readable instructions that when executed by a system cause the system to:

• configure a large language model with a conversational profile comprising training data that includes a set of example conversations that comprise labeled data demonstrating positive interactions, negative interactions, and appropriate responses and inappropriate responses, and a set of sentiments;
• receive a user input comprising a speech audio input;
• convert the user input into a text translation of the speech audio input;
• analyze the speech audio input to determine at least one of a volume, a tone of voice, and one or more inflections of a voice included in the speech audio input;
• analyze at least one of the volume, the tone of voice, and the one or more inflections with the text translation of the speech audio input using a prompt engine to determine a sentiment of the user input;
• generate a prompt, use the prompt engine, wherein the prompt includes the sentiment of the user input that is determined using at least one of the tone, the volume, the one or more inflections, and the text translation of the speech audio input, the prompt further comprising instructions to cause the large language model to include punctuation and a word selection in a text response that represents a selected sentiment from the set of sentiments within the conversational profile;
• cause the large language model to generate the text response based on the prompt and the text translation of the speech audio input, using the selected sentiment and the training data, wherein the training data causes the large language model to set patterns for subsequent text responses and prompts by increasing emphasis for positive user interactions and appropriate text responses and decreasing emphasis for negative user interactions and inappropriate text responses;
• receive the text response with the selected sentiment from the large language model generated from the prompt;
• select a style cue for an audio output based on the sentiment of the user input, the selected sentiment of the text response, and the prompt using the large language model;
• and
• generate the audio output using the text response and the style cue, wherein the audio output response uses the style cue, the punctuation, and the world selection from the text response to generate the audio output including at least one of an audio output volume, an audio output tone of voice, or one or more inflections of a voice in the audio output, wherein at least one of the audio output volume, the audio output tone of voice, or one or more inflections of the voice in the audio output corresponds to the style cue, the punctuation, and the word selection of the text response.