Emotive text-to-speech with auto detection of emotions

A computer-implemented method executed on data processing hardware that causes the data processing hardware to perform operations comprising:

• obtaining input text characterizing a natural language response generated by an assistant large language model (LLM) to a query input by a user during a conversation between the user and the assistant LLM, wherein the assistant LLM comprises a pre-trained LLM and a low-rank adaption training process fine-tunes a fraction of parameters of the pre-trained LLM to learn how to predict emotional states of input texts;
• processing, using the assistant LLM, the input text conditioned on an emotion detection task prompt to predict, as output from the assistant LLM, an emotional state of the natural language response, wherein the emotion detection task prompt specifies a task for the assistant LLM to detect an emotional state of the input text from a set of possible emotional states;
• determining, based on the emotional state of the natural language response predicted as output from the assistant LLM, an emotional embedding for the input text, the emotional embedding specifying the emotional state of the natural language response for synthesizing the input text into expressive speech;
• and
• instructing a text-to-speech (TTS) model to process the input text and the emotional embedding to generate a synthesized speech representation of the natural language response, the synthesized speech representation conveying the emotional state of the natural language response as specified by the emotional embedding,
• wherein the low-rank adaption training process fine-tunes the fraction of the pre-trained LLM by:
• receiving a training dataset of natural language training utterances, each natural language training utterance comprising:
• a corresponding textual representation of the natural language training utterance;
• and
• a corresponding ground-truth emotional state of the natural language training utterance;
• and
• for each natural language training utterance in the training dataset:
• processing, using the assistant LLM, the corresponding textual representation of the natural language training utterance to generate a corresponding predicted emotional state for the natural language training utterance as output from the assistant LLM;
• and
• determining a training loss based on the corresponding predicted emotional state and the corresponding ground-truth emotional state of the natural language training utterance;
• and
• fine-tuning, using the training losses, the fraction of the parameters of the assistant LLM while a remaining portion of the parameters of the assistant LLM are kept fixed.

A system comprising:

• data processing hardware;
• and
• memory hardware in communication with the data processing hardware, the memory hardware storing instructions that when executed on the data processing hardware cause the data processing hardware to perform operations comprising:
• obtaining input text characterizing a natural language response generated by an assistant large language model (LLM) to a query input by a user during a conversation between the user and the assistant LLM;
• receiving a fine-tuned prompt embedding, the fine-tuned prompt embedding comprising a soft prompt configured to guide the assistant LLM to detect an emotional state of the input text from a set of possible emotional states while parameters of the assistant LLM are held fixed;
• processing, using the assistant LLM, the input text conditioned on an emotion detection task prompt and the fine-tuned prompt embedding to predict, as output from the assistant LLM, an emotional state of the natural language response, wherein the emotion detection task prompt specifies a task for the assistant LLM to detect the emotional state of the input text from the set of possible emotional states;
• determining, based on the emotional state of the natural language response predicted as output from the assistant LLM, an emotional embedding for the input text, the emotional embedding specifying the emotional state of the natural language response for synthesizing the input text into expressive speech;
• and
• instructing a text-to-speech (TTS) model to process the input text and the emotional embedding to generate a synthesized speech representation of the natural language response, the synthesized speech representation conveying the emotional state of the natural language response as specified by the emotional embedding.

A system comprising:

• data processing hardware;
• and
• memory hardware in communication with the data processing hardware, the memory hardware storing instructions that when executed on the data processing hardware cause the data processing hardware to perform operations comprising:
• obtaining input text characterizing a natural language response generated by an assistant large language model (LLM) to a query input by a user during a conversation between the user and the assistant LLM, wherein the assistant LLM comprises a pre-trained LLM and a low-rank adaption training process fine-tunes a fraction of parameters of the pre-trained LLM to learn how to predict emotional states of input texts;
• processing, using the assistant LLM, the input text conditioned on an emotion detection task prompt to predict, as output from the assistant LLM, an emotional state of the natural language response, wherein the emotion detection task prompt specifies a task for the assistant LLM to detect an emotional state of the input text from a set of possible emotional states;
• determining, based on the emotional state of the natural language response predicted as output from the assistant LLM, an emotional embedding for the input text, the emotional embedding specifying the emotional state of the natural language response for synthesizing the input text into expressive speech;
• and
• instructing a text-to-speech (TTS) model to process the input text and the emotional embedding to generate a synthesized speech representation of the natural language response, the synthesized speech representation conveying the emotional state of the natural language response as specified by the emotional embedding,
• wherein the low-rank adaption training process fine-tunes the fraction of the pre-trained LLM by:
• receiving a training dataset of natural language training utterances, each natural language training utterance comprising:
• a corresponding textual representation of the natural language training utterance;
• and
• a corresponding ground-truth emotional state of the natural language training utterance;
• and
• for each natural language training utterance in the training dataset:
• processing, using the assistant LLM, the corresponding textual representation of the natural language training utterance to generate a corresponding predicted emotional state for the natural language training utterance as output from the assistant LLM;
• and
• determining a training loss based on the corresponding predicted emotional state and the corresponding ground-truth emotional state of the natural language training utterance;
• and
• fine-tuning, using the training losses, the fraction of the parameters of the assistant LLM while a remaining portion of the parameters of the assistant LLM are kept fixed.