# US 12,223,944 B2 · Dynamically adapting given assistant output based on a given persona assigned to an automated assistant > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,223,944 B2 - Document ID: US12223944B2 - Status: Granted - Kind: B2 - Application number: 17/744,440 - Filed: 2022-05-13 - Published: 2023-10-26 - Granted: 2025-02-11 - Earliest claimed: 2022-04-21 - Assignees: Google Inc. - Applicants: Google LLC - Inventors: Martin Baeuml; Thushan Amarasiriwardena; Roberto Pieraccini; Gianluca Martini - Web page: https://patentgrep.com/patent/US12223944B2 ## CPC - G10L 13/10: Prosody rules derived from text; Stress or intonation - G06F 3/167: {Audio in a user interface, e.g. using voice commands for navigating, audio feedback} - G06F 16/3329: Natural language query formulation - G06F 40/169: Annotation, e.g. comment data or footnotes - G06F 40/56: Natural language generation - G06N 3/006: based on simulated virtual individual or collective life forms, e.g. social simulations or particle swarm optimisation [PSO] - G06N 3/044: Recurrent networks, e.g. Hopfield networks - G06N 3/045: Combinations of networks - G06N 3/08: Learning methods - G06N 3/084: Backpropagation, e.g. using gradient descent - G06N 5/022: Knowledge engineering; Knowledge acquisition - G06N 7/01: Probabilistic graphical models, e.g. probabilistic networks - G06N 20/00: Machine learning - G06N 20/20: Ensemble learning - G06T 7/20: Analysis of motion (motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51) - G06V 20/40: in video content (extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44) - G06V 40/20: Movements or behaviour, e.g. gesture recognition (recognition of facial expressions G06V40/16) - G10L 13/02: Methods for producing synthetic speech; Speech synthesisers - G10L 13/08: Text analysis or generation of parameters for speech synthesis out of text, e.g. grapheme to phoneme translation, prosody generation or stress or intonation determination - G10L 15/063: {Training} - G10L 15/1815: {Semantic context, e.g. disambiguation of the recognition hypotheses based on word meaning} - G10L 15/183: using context dependencies, e.g. language models - G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue - G10L 25/57: for processing of video signals - H04N 5/04: Synchronising (for television systems using pulse code modulation H04N7/56) - G06T 2207/10016: Video; Image sequence - G06T 2207/30196: Human being; Person - G10L 13/033: Voice editing, e.g. manipulating the voice of the synthesiser - G10L 2015/223: {Execution procedure of a spoken command} - G10L 2015/228: {of application context} - G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract Implementations relate to dynamically adapting a given assistant output based on a given persona, from among a plurality of disparate personas, assigned to an automated assistant. In some implementations, the given assistant output can be generated and subsequently adapted based on the given persona assigned to the automated assistant. In other implementations, the given assistant output can be generated specific to the given persona and without having to subsequently adapt the given assistant output to the given persona. Notably, the given assistant output can include a stream of textual content to be synthesized for audible presentation to the user, and a stream of visual cues utilized in controlling a display of a client device and/or in controlling a visualized representation of the automated assistant. Various implementations utilize large language models (LLMs), or output previously generated utilizing LLMs, to reflect the given persona in the given assistant output. ## Claims ### Claim 1 A method implemented by one or more processors, the method comprising: obtaining, from an online multimedia repository, video content that includes a stream of audio data for audible content of the video content and a stream of vision data for visual content of the video content; processing, using an automatic speech recognition model, the stream of audio data for the audible content of the video content to generate a stream of textual content corresponding to one or more spoken utterances captured in the stream of audio data for the audible content of the video content; processing, using one or more movement tracking machine learning models, the stream of vision data for the visual content of the video content to generate a stream of visual cues corresponding to one or more movements captured in the stream of vision data for the visual content of the video content; generating, based on processing the stream of audio data and based on processing the stream of video data, a given persona training data instance to be utilized in further training an instance of a given large language model (LLM) that is specific to a given persona, from among of a plurality of disparate personas, embodied in the video content; training the instance of the given LLM based on at least the given persona training instance; and causing the instance of the given LLM to be utilized in subsequently processing additional streams of audio data capturing additional spoken utterances directed to an instance of an automated assistant that is assigned the given persona. ### Claim 2 (depends on 1) The method of claim 1, wherein the online multimedia repository is a public online multimedia repository that enables a plurality of users to upload and share corresponding video content with other users. ### Claim 3 (depends on 1) The method of claim 1, wherein the stream of audio data for the audible content of the video content captures speech of a person or character associated with the given persona embodied in the video content. ### Claim 4 (depends on 3) The method of claim 3, wherein the stream of vision data for the visual content of the video content captures movements or gestures of the person or character associated with the given persona embodied in the video content as the one or more spoken utterances captured in the stream of audio data for the audible content of the video content are provided by the person or character. ### Claim 5 (depends on 1) The method of claim 1, wherein the stream of audio data for the audible content of the video content and the stream of vision data for the visual content of the video content are synchronized via one or more video timestamps. ### Claim 6 (depends on 5) The method of claim 5, wherein training the instance of the given LLM based on at least the given persona training instance comprises: annotating, based on the one or more video timestamps, the stream of textual context with one or more visual cue timestamps that indicate when the stream of visual cues is to be utilized in controlling a display of a client device and/or for controlling a visualized representation of the instance of the automated assistant. ### Claim 7 (depends on 6) The method of claim 6, wherein the one or more visual cue timestamps include at least a start visual cue timestamp that indicates when a given visual cue, included in the stream of visual cues, will start being utilized in controlling the display of the client device and/or for controlling the visualized representation of the instance of the automated assistant, and a stop visual cue timestamp that indicates when the given visual cue, included in the stream of visual cues, will stop being utilized in controlling the display of the client device and/or for controlling the visualized representation of the instance of the automated assistant. ### Claim 8 (depends on 1) The method of claim 1, wherein the one or more movement tracking machine learning models comprise: an eye gaze machine learning model trained to detect eye movement that is captured in the stream of vision data for the visual content of the video content; a mouth movement machine learning model trained to detect mouth movement that is captured in the stream of vision data for the visual content of the video content; a lip movement machine learning model trained to detect lip movement that is captured in the stream of vision data for the visual content of the video content; a body movement machine learning model trained to detect body movement that is captured in the stream of vision data for the visual content of the video content; and/or a body posture machine learning model trained to detect body posture that is captured in the stream of vision data for the visual content of the video content. ### Claim 9 (depends on 1) The method of claim 1, wherein training the instance of the given LLM based on at least the given persona training instance comprises: applying training instance input, of the given persona training instance, as input across the given LLM to generate predicted output, the training instance input including the stream of audio data for the audible content of the video content and/or the stream of vision data for the visual content of the video content, and the predicted output including a predicted stream of textual content and/or a predicted stream of visual cues; comparing the predicted output to training instance output, of the given persona training instance, to generate one or more losses, the training instance output including the stream of textual content corresponding to one or more spoken utterances captured in the stream of audio data for the audible content of the video content and/or the stream of visual cues corresponding to one or more movements captured in the stream of vision data for the visual content of the video content; and causing the given LLM to be updated based on the one or more losses. ### Claim 10 (depends on 9) The method of claim 9, wherein comparing the predicted output to the training instance output to generate the one or more losses comprises: generating a first loss, of the one or more losses, based on comparing the predicted stream of textual content, of the predicted output, to the stream of textual content corresponding to one or more spoken utterances captured in the stream of audio data for the audible content of the video content. ### Claim 11 (depends on 10) The method of claim 10, wherein comparing the predicted output to the training instance output to generate the one or more losses further comprises: generating a second loss, of the one or more losses, based on comparing the predicted stream of visual cues, of the predicted output, to the stream of visual cues corresponding to one or more movements captured in the stream of vision data for the visual content of the video content. ### Claim 12 (depends on 1) The method of claim 1, wherein causing the instance of the given LLM to be utilized in subsequently processing the additional audio data capturing the additional spoken utterances directed to the instance of the automated assistant that is assigned the given persona comprises: receiving a given additional stream of audio data that captures a given additional spoken utterance of a user of a client device, the given additional stream of audio data being generated by one or more microphones of the client device, and the given additional spoken utterance being directed to the instance of the automated assistant executing at least in part at the client device; generating, based on processing the given additional stream of audio data and using the instance of the given LLM, a given assistant output that is responsive to the spoken utterance and that is specific to the given persona, wherein the given assistant output includes: (i) a given additional stream of textual content; and (ii) a given additional stream of visual cues for controlling a display of the client device responsive to the given additional spoken utterance and/or for controlling a visualized representation of the instance of the automated assistant that is visually rendered for presentation to the user via the display of the client device; and in response to receiving the given additional stream of audio data that captures the given additional spoken utterance of the user of the client device: causing synthesized speech audio data capturing synthesized speech corresponding to the given additional stream of textual context to be audibly rendered for presentation to the user via one or more speakers of the client device; and causing the given additional stream of visual cues to be utilized in controlling the display of the client device and/or in controlling the visualized representation of the instance of the automated assistant. ### Claim 13 (depends on 12) The method of claim 12, further comprising: synchronizing, for presentation to the user, the audible rendering of the synthesized speech corresponding to the given additional stream of textual context and the utilization of the given additional stream of visual cues in controlling the display of the client device and/or in controlling the visualized representation of the instance of the automated assistant. ### Claim 14 (depends on 13) The method of claim 13, wherein the audible rendering of the synthesized speech corresponding to the given additional stream of textual context and the utilization of the given additional stream of visual cues in controlling the display of the client device and/or in controlling the visualized representation of the instance of the automated assistant are synchronized based on one or more visual cue timestamps. ### Claim 15 A system comprising: one or more processors; and memory storing instructions that, when executed, cause the one or more processors to perform operations, the operations comprising: obtaining, from an online multimedia repository, video content that includes a stream of audio data for audible content of the video content and a stream of vision data for visual content of the video content; processing, using an automatic speech recognition model, the stream of audio data for the audible content of the video content to generate a stream of textual content corresponding to one or more spoken utterances captured in the stream of audio data for the audible content of the video content; processing, using one or more movement tracking machine learning models, the stream of vision data for the visual content of the video content to generate the stream of visual cues; generating, based on processing the stream of audio data and based on processing the stream of video data, a given persona training data instance to be utilized in further training an instance of a given large language model (LLM) that is specific to a given persona, from among of a plurality of disparate personas, embodied in the video content; training the instance of the given LLM based on at least the given persona training instance; and causing the instance of the given LLM to be utilized in subsequently processing additional audio data capturing additional spoken utterances directed to an instance of an automated assistant that is assigned the given persona. ### Claim 16 (depends on 15) The system of claim 15, wherein the stream of audio data for the audible content of the video content captures speech of a person or character associated with the given persona embodied in the video content. ### Claim 17 (depends on 15) The system of claim 15, wherein the stream of audio data for the audible content of the video content and the stream of vision data for the visual content of the video content are synchronized via one or more video timestamps. ### Claim 18 (depends on 15) The system of claim 15, wherein the one or more movement tracking machine learning models comprise: an eye gaze machine learning model trained to detect eye movement that is captured in the stream of vision data for the visual content of the video content; a mouth movement machine learning model trained to detect mouth movement that is captured in the stream of vision data for the visual content of the video content; a lip movement machine learning model trained to detect lip movement that is captured in the stream of vision data for the visual content of the video content; a body movement machine learning model trained to detect body movement that is captured in the stream of vision data for the visual content of the video content; and/or a body posture machine learning model trained to detect body posture that is captured in the stream of vision data for the visual content of the video content. ### Claim 19 A non-transitory computer-readable storage medium storing instructions that, when executed, cause at least one processor to perform operations, the operations comprising: obtaining, from an online multimedia repository, video content that includes a stream of audio data for audible content of the video content and a stream of vision data for visual content of the video content; processing, using an automatic speech recognition model, the stream of audio data for the audible content of the video content to generate a stream of textual content corresponding to one or more spoken utterances captured in the stream of audio data for the audible content of the video content; processing, using one or more movement tracking machine learning models, the stream of vision data for the visual content of the video content to generate a stream of visual cues corresponding to one or more movements captured in the stream of vision data for the visual content of the video content; generating, based on processing the stream of audio data and based on processing the stream of video data, a given persona training data instance to be utilized in further training an instance of a given large language model (LLM) that is specific to a given persona, from among of a plurality of disparate personas, embodied in the video content; training the instance of the given LLM based on at least the given persona training instance; and causing the instance of the given LLM to be utilized in subsequently processing additional streams of audio data capturing additional spoken utterances directed to an instance of an automated assistant that is assigned the given persona. ### Claim 20 (depends on 15) The system of claim 15, wherein the online multimedia repository is a public online multimedia repository that enables a plurality of users to upload and share corresponding video content with other users.