# US 2024/0205174 A1 · Device sensor information as context for interactive chatbot > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2024/0205174 A1 - Document ID: US20240205174A1 - Status: Published application - Kind: A1 - Application number: 18/081,541 - Filed: 2022-12-14 - Published: 2024-06-20 - Earliest claimed: 2022-12-14 - Assignees: Google Inc. - Applicants: Google LLC - Inventors: Alexander Bailey - Web page: https://patentgrep.com/patent/US20240205174A1 ## CPC - G06F 40/35: Discourse or dialogue representation - H04L 51/02: using automatic reactions or user delegation, e.g. automatic replies or chatbot-generated messages - G06F 40/30: Semantic analysis - G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU] - G06N 3/045: Combinations of networks - G10L 13/027: Concept to speech synthesisers; Generation of natural phrases from machine-based concepts (generation of parameters for speech synthesis out of text G10L13/08) - G10L 13/0335: {Pitch control} - G10L 13/08: Text analysis or generation of parameters for speech synthesis out of text, e.g. grapheme to phoneme translation, prosody generation or stress or intonation determination - G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue - H04L 51/06: Message adaptation to terminal or network requirements - G06F 3/167: {Audio in a user interface, e.g. using voice commands for navigating, audio feedback} - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING ## Abstract Implementations relate to processing, utilizing a large language model (“LLM”), input that is based on sensor data, from sensor(s) of a client device, to generate LLM output—and causing output, that is based on the generated LLM output, to be rendered by an interactive chatbot. The input that is based on sensor data and that is processed by the LLM in generating the LLM output can be, or can include, non-acoustic input based on non-acoustic sensor data. For example, an instance of LLM output can be generated based on processing of non-acoustic input using the LLM and without any processing of acoustic input (that is based on acoustic sensor data) using the LLM. As another example, an instance of LLM output can be generated based on processing, using the LLM, both non-acoustic input that is based on non-acoustic data and acoustic input that is based on acoustic sensor data. ## Claims ### Claim 1 A method implemented by one or more processors, the method comprising: receiving non-acoustic sensor data from a non-acoustic sensor of a client device; determining whether the received non-acoustic sensor data satisfies one or more conditions; and in response to determining that the received non-acoustic sensor data satisfies the one or more conditions: processing the received non-acoustic sensor data to generate a natural language description for the non-acoustic sensor data, processing, using a large language model (LLM), the generated natural language description for the non-acoustic sensor data, as input, to generate an LLM output, generating, based on the LLM output, a natural language statement that is responsive to the received non-acoustic sensor data, and causing the natural language statement to be rendered at the client device via an interactive chatbot that is installed at the client device or that is otherwise accessible via the client device. ### Claim 2 (depends on 1) The method of claim 1, further comprising: tuning, based on the LLM output, a voice of a virtual character that visually represents the interactive chatbot. ### Claim 3 (depends on 2) The method of claim 2, wherein causing the natural language statement to be rendered at the client device via the chatbot comprises: causing the natural language statement to be audibly rendered, in the tuned voice of the virtual character, via an audible interface of the client device. ### Claim 4 (depends on 1) The method of claim 1, further comprising: modifying, based on the LLM output, a visual appearance of a graphical interface of the interactive chatbot, wherein modifying the visual appearance of the graphical interface of the interactive chatbot comprises: modifying a character visual appearance of a virtual character displayed at the graphical interface of the interactive chatbot; and/or modifying a background of the graphical interface of the interactive chatbot. ### Claim 5 (depends on 4) The method of claim 4, wherein modifying the visual appearance of the graphical interface of the interactive chatbot comprises modifying the character visual appearance of the virtual character, and wherein modifying the character visual appearance of the virtual character comprises: controlling, based on the LLM output, a facial expression, a gesture, and/or a movement of the virtual character. ### Claim 6 (depends on 1) The method of claim 1, wherein processing the received non-acoustic sensor data to generate the natural language description responsive to the non-acoustic sensor data comprises: determining, based on the received non-acoustic sensor data, a client device state of the client device or an environment state of an environment of the client device, and generating the natural language description to reflect the client device state or the environment state of the client device. ### Claim 7 (depends on 1) The method of claim 1, wherein determining whether the non-acoustic received sensor data satisfies the one or more conditions comprises: determining whether a numerical value, in the received non-acoustic sensor data detected by the non-acoustic sensor, satisfies a threshold value; determining that the received non-acoustic sensor data satisfies the one or more conditions based on determining that the numerical value satisfies the threshold value; and determining that the received non-acoustic sensor data does not satisfy the one or more conditions based on determining that the numerical value does not satisfy the threshold value. ### Claim 8 (depends on 1) The method of claim 1, wherein determining whether the non-acoustic received sensor data satisfies the one or more conditions comprises: determining, based on a particular virtual character being a currently active virtual character for the interactive chatbot, whether the received non-acoustic sensor data is of a type for which the particular virtual character is responsive; determining that the received non-acoustic sensor data satisfies the one or more conditions based on determining that the received non-acoustic sensor data is of the type for which the particular virtual character is responsive; and determining that the received non-acoustic sensor data fails to satisfy the one or more conditions based on determining that the received non-acoustic sensor data is not of the type for which the particular virtual character is responsive. ### Claim 9 (depends on 1) The method of claim 1, wherein determining whether the received non-acoustic sensor data satisfies one or more conditions comprises: determining, based on a particular virtual character being a currently active virtual character for the interactive chatbot, whether the received non-acoustic sensor data includes content for which the particular virtual character is responsive; determining that the received non-acoustic sensor data satisfies the one or more conditions based on determining that the received non-acoustic sensor data includes content for which the particular virtual character is responsive; and determining that the received non-acoustic sensor data fails to satisfy the one or more conditions based on determining that the received non-acoustic sensor data does not include content for which the particular virtual character is responsive. ### Claim 10 (depends on 1) The method of claim 1, wherein the one or more conditions are specific to a current configuration, for one or more adjustable settings, of the interactive chatbot and for the client device. ### Claim 11 (depends on 10) The method of claim 10, wherein the one or more conditions are specific to a type, service, or function of the one or more adjustable settings of the interactive chatbot. ### Claim 12 (depends on 1) The method of claim 1, further comprising: in response to determining that the received non-acoustic sensor data does not satisfy the one or more conditions: bypassing processing of the received non-acoustic sensor data to generate the natural language description, and/or bypassing processing, using the LLM, the generated natural language description. ### Claim 13 (depends on 12) The method of claim 12, further comprising: in response to determining that the received non-acoustic sensor data does not satisfy the one or more conditions: discarding the received non-acoustic sensor data without performing any further processing of the received non-acoustic sensor data. ### Claim 14 (depends on 1) The method of claim 1, wherein causing the natural language statement to be rendered at the client device via the interactive chatbot comprises: causing the natural language statement to be visually rendered via a graphical interface of the interactive chatbot. ### Claim 15 (depends on 1) The method of claim 1, further comprising: receiving audio data from an acoustic sensor of the client device; performing speech recognition, based on the audio data, to generate recognized natural language content recognized from a spoken utterance captured by the audio data; and processing, using the LLM and along with the generated natural language description for the non-acoustic sensor data, the recognized natural language content to generate the LLM output. ### Claim 16 (depends on 15) The method of claim 15, wherein processing the generated natural language description for the non-acoustic sensor data along with the recognized content using the LLM is in response to the audio data and the non-acoustic sensor data being received in a same human-to-computer dialog and/or being received within a threshold period of time of one another. ### Claim 17 (depends on 15) The method of claim 15, wherein processing the generated natural language description for the non-acoustic sensor data along with the recognized content using the LLM comprises: priming the LLM by processing, using the LLM, the generated natural language description for the non-acoustic sensor data; and processing, using the LLM and after priming the LLM, the recognized natural language content to generate the LLM output. ### Claim 18 (depends on 1) The method of claim 1, further comprising: processing, using the LLM and along with the generated natural language description for the non-acoustic sensor data, context data for an ongoing human-to-computer dialog between a user of the client device and the interactive chatbot. ### Claim 19 (depends on 18) The method of claim 18, wherein the context data includes a current utterance from the user in the ongoing human-to-computer dialog, a prior utterance from the user in the ongoing human-to-computer dialog, a current response from the interactive chatbot in the ongoing human-to-computer dialog, and/or a prior response from the interactive chatbot in the ongoing human-to-computer dialog. ### Claim 20 (depends on 18) The method of claim 18, wherein the context data includes a current date, a current time, and/or a current day of the week. ### Claim 21 A method implemented by one or more processors, the method comprising: receiving non-acoustic sensor data to which an interactive chatbot is responsive, wherein the interactive chatbot is installed at or accessible via a client device; processing the received non-acoustic sensor data to generate a natural language description for the non-acoustic sensor data; processing, using a large language model (LLM), the generated natural language description for the non-acoustic sensor data to generate an LLM output; generating, based on the LLM output, a natural language statement that is responsive to the received non-acoustic sensor data; tuning, based on the LLM output, a voice of the interactive chatbot; and causing the natural language statement to be audibly rendered in the tuned voice of the interactive chatbot. ### Claim 22 A method implemented by one or more processors, the method comprising: receiving audio data from one or more acoustic sensors of a client device; receiving non-acoustic audio data from one or more non-acoustic sensors of the client device; processing, using a large language model (LLM), input generated based on both the audio data and the non-acoustic audio data, to generate an LLM output; generating, based on the LLM output, chatbot output for an interactive chatbot; and causing the chatbot output to be rendered by the interactive chatbot and at the client device.