Controlling a style of large language model(s) during ongoing dialog(s) through utilization of natural language based response style tag(s)

A method implemented by one or more processors, the method comprising:

• as part of an ongoing spoken dialog between a user of a client device and an automated assistant that is accessible at the client device:
• receiving spoken natural language (NL) based input from the user of the client device during a given dialog turn of the ongoing spoken dialog between the user of the client device and the automated assistant;
• determining, based on a first set of prosodic properties of the spoken NL based input, one or more style signals that are specific to the given dialog turn of the ongoing spoken dialog between the user of the client device and the automated assistant;
• determining, using a large language model (LLM), and based on the one or more style signals, a given NL based response style, from among a plurality of disparate NL based response styles, that is not specified by the spoken NL based input but is to be utilized in responding to the spoken NL based input, wherein the LLM was previously fine-tuned with respect to the plurality of disparate NL based response styles prior to the ongoing spoken dialog between the user of a client device and the automated assistant that is accessible at the client device;
• processing, using the LLM, the spoken NL based input and a given NL based response style tag that is associated with the given NL based response style to generate LLM output, wherein the LLM output comprises a probability distribution over a sequence of words or phrases;
• determining, based on the LLM output, an audible NL based response that is in the given NL based response style and that is responsive to the spoken NL based input, wherein determining the audible NL based response that is in the given NL based response style and that is responsive to the spoken NL based input based on the LLM output comprises:
• biasing, based on the given NL based response style, selection of one or more of the words or phrases, from the LLM output, for inclusion in the audible NL based response;
• and
• causing the audible NL based response to be rendered at the client device, wherein the audible NL based response is synthesized using a second set of prosodic properties that are associated with the given NL based response style, and wherein the given NL based response style is dynamically adapted between two or more of the plurality of disparate NL based response styles throughout the ongoing spoken dialog.

A system comprising:

• at least one processor;
• and
• memory storing instructions that, when executed by the at least one processor, cause the at least one processor to be operable to:
• as part of an ongoing spoken dialog between a user of a client device and an automated assistant that is accessible at the client device:
• receive spoken natural language (NL) based input from the user of the client device during a given dialog turn of the ongoing spoken dialog between the user of the client device and the automated assistant;
• determine, based on a first set of prosodic properties of the spoken NL based input, one or more style signals that are specific to the given dialog turn of the ongoing dialog between the user of the client device and the automated assistant;
• process, using a large language model (LLM) behavior controller, the one or more style signals to determine a given NL based response style, from among a plurality of disparate NL based response styles, that is not specified by the spoken NL based input but is to be utilized in responding to the spoken NL based input;
• process, using the LLM, the spoken NL based input and a given NL based response style tag that is associated with the given NL based response style to generate LLM output, wherein the LLM output comprises a probability distribution over a sequence of words or phrases;
• determine, based on the LLM output, an audible NL based response that is in the given NL based response style and that is responsive to the spoken NL based input, wherein the instructions to determine the audible NL based response that is in the given NL based response style and that is responsive to the spoken NL based input based on the LLM output comprise instructions to:
• bias, based on the given NL based response style, selection of one or more of the words or phrases, from the LLM output, for inclusion in the audible NL based response;
• and
• cause the audible NL based response to be rendered at the client device, wherein the audible NL based response is synthesized using a second set of prosodic properties that are associated with the given NL based response style, and wherein the given NL based response style is dynamically adapted between two or more of the plurality of disparate NL based response styles throughout the ongoing spoken dialog.