# US 12,190,879 B2 · Controlling device output according to a determined condition of a user > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,190,879 B2 - Document ID: US12190879B2 - Status: Granted - Kind: B2 - Application number: 17/891,319 - Filed: 2022-08-19 - Published: 2022-12-22 - Granted: 2025-01-07 - Earliest claimed: 2018-11-19 - Assignees: Google Inc. - Applicants: Google LLC - Inventors: Tiruvilwamalai Raman - Web page: https://patentgrep.com/patent/US12190879B2 ## CPC - G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue - G16H 20/70: relating to mental therapies, e.g. psychological therapy or autogenous training - A61B 5/0024: {for multiple sensor units attached to the patient, e.g. using a body or personal area network} - A61M 21/02: for inducing sleep or relaxation, e.g. by direct nerve stimulation, hypnosis, analgesia (for massage A61H; electrotherapy A61N, e.g. applying alternating or intermittent electric currents for producing anaesthesia A61N1/36021) - G16H 40/63: for local operation - G16H 50/30: for calculating health indices; for individual health risk assessment - H04N 21/42203: {sound input device, e.g. microphone} - H04N 21/44218: {Detecting physical presence or behaviour of the user, e.g. using sensors to detect if the user is leaving the room or changes his face expression during a TV programme (methods or arrangements for recognising human body or animal bodies or body parts G06V40/10; methods or arrangements for acquiring or recognising human faces, facial parts, facial sketches, facial expressions G06V40/16; methods or arrangements for recognising movements or behaviour G06V40/20; arrangements for identifying users in broadcast systems H04H60/45)} - H04N 21/4852: {for modifying audio parameters, e.g. switching between mono and stereo} - H04N 21/4854: {for modifying image parameters, e.g. image brightness, contrast} - H04N 21/8456: {by decomposing the content in the time domain, e.g. in time segments} - A61M 2021/0027: {by the hearing sense} - A61M 2021/0044: {by the sight sense} - A61M 2021/005: {images, e.g. video} - A61M 2021/0083: {especially for waking up} - A61M 2205/3584: using modem, internet or Bluetooth® - A61M 2205/3592: using telemetric means, e.g. radio or optical transmission - A61M 2205/502: User interfaces, e.g. screens or keyboards - A61M 2205/80: voice-operated command - A61M 2209/088: on the body - A61M 2230/06: Heartbeat rate only - A61M 2230/205: partial oxygen pressure (P-O2) - A61M 2230/30: Blood pressure (A61M2230/04 takes precedence) - A61M 2230/42: Rate - A61M 2230/63: Motion, e.g. physical activity - G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING - G16H: HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA - A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D) - A61M: DEVICES FOR INTRODUCING MEDIA INTO, OR ONTO, THE BODY (introducing media into or onto the bodies of animals A61D7/00; means for inserting tampons A61F13/26; devices for administering food or medicines orally A61J; containers for collecting, storing or administering blood or medical fluids A61J1/05); DEVICES FOR TRANSDUCING BODY MEDIA OR FOR TAKING MEDIA FROM THE BODY (surgery A61B; chemical aspects of surgical articles A61L); DEVICES FOR PRODUCING OR ENDING SLEEP OR STUPOR {(Electrotherapy, e.g. producing anaesthesia by the use of alternating or intermittent currents A61N1/36021)} - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract Dynamically controlling output from a device, such as an automated assistant device. Control of the output can be based on, for example, a condition and/or physiological attribute(s) of a user of the device. Various implementations dynamically control the output to improve sleep quality for the user and/or mitigate waste of computational and/or network resources. ## Claims ### Claim 1 A method, implemented by one or more processors of an assistant computing device, comprising: in response to detecting an occurrence of an invocation at the assistant computing device: transitioning the assistant computing device from a first state to a second state, wherein in the first state the assistant computing device does not perform certain processing of audio data detected via one or more microphones of the assistant computing device, and wherein in the second state the assistant computing device performs the certain processing of audio data detected via the one or more microphones; responsive to transitioning the assistant computing device to the second state, and for a duration of the second state: rendering ambient audio via one or more speakers of the assistant computing device, wherein the rendering is for the duration of the second state in which the assistant computing device performs the certain processing of audio data, and wherein the duration of the second state begins with transitioning the assistant computing device from the first state to the second state, in which the automated assistant performs the certain processing of audio data, and wherein the duration of the second state ends with the cessation of the second state; and responsive to cessation of the second state: ceasing the rendering of the ambient audio. ### Claim 2 (depends on 1) The method of claim 1, further comprising: when in the first state: monitoring for occurrence of an invocation phrase by processing audio data detected via the one or more microphones using only a local invocation model stored locally at the assistant computing device, wherein transitioning the assistant computing device from the first state to the second state is based on detecting the occurrence of the invocation phrase in a portion of the audio data. ### Claim 3 (depends on 1) The method of claim 1, further comprising: when in the first state: monitoring for occurrence of an invocation gesture by processing video data detected via one or more cameras of the assistant computing device, wherein transitioning the assistant computing device from the first state to the second state is based on detecting the occurrence of the invocation gesture in a portion of the video data. ### Claim 4 (depends on 1) The method of claim 1, further comprising: selecting the ambient audio based on the ambient audio being stored in association with a user profile that is determined based at least in part on further analysis of the portion of the audio data. ### Claim 5 (depends on 1) The method of claim 1, further comprising: determining a volume for the rendering of the ambient audio, wherein rendering the ambient audio is at the determined volume. ### Claim 6 (depends on 5) The method of claim 5, wherein determining the volume is based on one or more of: a level of background noise as detected via the one or more microphones, an estimated distance of an active user of the assistant computing device, or one or more physiological characteristics of the active user. ### Claim 7 (depends on 1) The method of claim 1, wherein the certain processing comprises one or more of: transmitting the audio data to a remote automated assistant server and/or speech-to-text processing. ### Claim 8 (depends on 1) The method of claim 1, wherein, in the first state: rendering a response that is responsive to a previous request received via the assistant device, and wherein transitioning the assistant computing device from the first state to the second state occurs automatically in response to completion of rendering the response. ### Claim 9 A method, implemented by one or more processors of an assistant computing device, comprising: monitoring for occurrence of an invocation phrase by processing audio data detected via the one or more microphones; in response to detecting the invocation phrase in a portion of the audio data: rendering ambient audio via one or more speakers of the assistant computing device; performing, while rendering the ambient audio, certain processing, of audio data detected via the one or more microphones, that is not performed prior to detecting the invocation phrase, wherein the certain processing includes processing a request that is included in the audio data, and wherein performing the certain processing of the audio data begins when rendering the ambient audio begins and ends when rendering the ambient audio ends; and rendering a response that is responsive to the request; and in response to starting rendering of the response: ceasing the rendering of the ambient audio. ### Claim 10 (depends on 9) The method of claim 9, wherein monitoring for occurrence of the invocation phrase by processing audio data detected via the one or more microphones includes using only a local invocation model stored locally at the assistant computing device. ### Claim 11 (depends on 9) The method of claim 9, wherein the certain processing comprises speech-to-text processing. ### Claim 12 (depends on 9) The method of claim 9, further comprising: processing the audio data to determine a user profile for a user that uttered the invocation phrase; and selecting the ambient audio based on the ambient audio being stored in association with the user profile. ### Claim 13 (depends on 9) The method of claim 9, further comprising: determining a volume for the rendering of the ambient audio, wherein rendering the ambient audio is at the determined volume. ### Claim 14 (depends on 13) The method of claim 13, wherein determining the volume is based on one or more of: a level of background noise as detected via the one or more microphones, an estimated distance of an active user of the assistant computing device, or one or more physiological characteristics of the active user. ### Claim 15 An assistant computing device, comprising one or more processors, and memory configured to store instructions that, when executed by one or more of the processors, cause one or more of the processors to perform operations that include: in response to detecting an occurrence of an invocation at the assistant computing device: transitioning the assistant computing device from a first state to a second state, wherein in the first state the assistant computing device does not perform certain processing of audio data detected via one or more microphones of the assistant computing device, and wherein in the second state the assistant computing device performs the certain processing of audio data detected via the one or more microphones; responsive to transitioning the assistant computing device to the second state, and for a duration of the second state: rendering ambient audio via one or more speakers of the assistant computing device, wherein the rendering is for the duration of the second state in which the assistant computing device performs the certain processing of audio data, and wherein the duration of the second state begins with transitioning the assistant computing device from the first state to the second state, in which the automated assistant performs the certain processing of audio data, and wherein the duration of the second state ends with the cessation of the second state; and responsive to cessation of the second state: ceasing the rendering of the ambient audio. ### Claim 16 (depends on 15) The computing device of claim 15, wherein the operations further include: when in the first state: monitoring for occurrence of an invocation phrase by processing audio data detected via the one or more microphones using only a local invocation model stored locally at the assistant computing device, wherein transitioning the assistant computing device from the first state to the second state is based on detecting the occurrence of the invocation phrase in a portion of the audio data. ### Claim 17 (depends on 15) The computing device of claim 15, wherein the operations further include: selecting the ambient audio based on the ambient audio being stored in association with a user profile that is determined based at least in part on further analysis of the portion of the audio data. ### Claim 18 (depends on 15) The computing device of claim 15, wherein the operations further include: determining a volume for the rendering of the ambient audio, wherein rendering the ambient audio is at the determined volume. ### Claim 19 (depends on 18) The computing device of claim 18, wherein determining the volume is based on one or more of: a level of background noise as detected via the one or more microphones, an estimated distance of an active user of the assistant computing device, or one or more physiological characteristics of the active user. ### Claim 20 (depends on 15) The computing device of claim 15, wherein the certain processing comprises one or more of: transmitting the audio data to a remote automated assistant server, or speech-to-text processing.