# US 2023/0053873 A1 · Invoking automated assistant function(s) based on detected gesture and gaze > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2023/0053873 A1 - Document ID: US20230053873A1 - Status: Published application - Kind: A1 - Application number: 17/981,181 - Filed: 2022-11-04 - Published: 2023-02-23 - Earliest claimed: 2019-10-18 - Assignees: Google Inc. - Applicants: Google LLC - Inventors: Yuan Yuan; Kenneth Mixter; Tuan Nguyen - Web page: https://patentgrep.com/patent/US20230053873A1 ## CPC - G06F 3/013: {Eye tracking input arrangements (G06F3/015 takes precedence)} - G06F 3/0304: {Detection arrangements using opto-electronic means (constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042)} - G06F 3/038: Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry - G06F 3/167: {Audio in a user interface, e.g. using voice commands for navigating, audio feedback} - G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue - G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)} - G06F 3/16: Sound input; Sound output (speech processing G10L) - G06F 2203/0381: Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING ## Abstract Invoking one or more previously dormant functions of an automated assistant in response to detecting, based on processing of vision data from one or more vision components: (1) a particular gesture (e.g., of one or more “invocation gestures”) of a user; and/or (2) detecting that a gaze of the user is directed at an assistant device that provides an automated assistant interface (graphical and/or audible) of the automated assistant. For example, the previously dormant function(s) can be invoked in response to detecting the particular gesture, detecting that the gaze of the user is directed at an assistant device for at least a threshold amount of time, and optionally that the particular gesture and the directed gaze of the user co-occur or occur within a threshold temporal proximity of one another. ## Claims ### Claim 1 A client device comprising: at least one vision component; at least one microphone; one or more processors; memory operably coupled with the one or more processors, wherein the memory stores instructions that, in response to execution of the instructions by one or more of the processors, cause one or more of the processors to perform the following operations: receiving a stream of vision data that is based on output from the vision component of the client device; receiving a stream of audio data that is based on output from the microphone of the client device; determining, based on processing the vision data: that a gaze of a user is directed toward the client device, and a user profile for the user; determining, based on processing the audio data, that a spoken utterance, included in the audio data: temporally corresponds to the gaze, and has voice characteristics that match the user profile that is determined based on processing the vision data; and in response to determining the gaze of the user, and contingent on determining that the spoken utterance temporally corresponds to the gaze and has the voice characteristics that match the user profile that is determined based on processing the vision data: causing at least one dormant function of the automated assistant to be activated. ### Claim 2 (depends on 1) The client device of claim 1, wherein the at least one dormant function of the automated assistant, that is caused to be activated in response to determining the gaze of the user, and contingent on determining that the spoken utterance temporally corresponds to the gaze and has the voice characteristics that match the user profile that is determined based on processing the vision data comprises: transmitting of data, from the client device, to a remote server associated with the automated assistant. ### Claim 3 (depends on 1) The client device of claim 1, wherein the at least one dormant function of the automated assistant, that is caused to be activated in response to determining the gaze of the user, and contingent on determining that the spoken utterance temporally corresponds to the gaze and has the voice characteristics that match the user profile that is determined based on processing the vision data further comprises: graphically rendering content that is tailored to the user profile. ### Claim 4 (depends on 1) The client device of claim 1, wherein the at least one dormant function of the automated assistant, that is caused to be activated in response to determining the gaze of the user, and contingent on determining that the spoken utterance temporally corresponds to the gaze and has the voice characteristics that match the user profile that is determined based on processing the vision data comprises: automatic speech processing of the audio data. ### Claim 5 (depends on 1) The client device of claim 1, wherein determining, based on processing the vision data, the user profile of the user comprises performing facial recognition based on processing the vision data. ### Claim 6 (depends on 1) The client device of claim 1, wherein determining, based on processing the vision data, that the gaze of the user is directed toward the client device comprises processing the vision data using a trained gaze machine learning model stored locally at the client device. ### Claim 7 (depends on 1) The client device of claim 1, further comprising: determining that the user profile is authorized for the client device; wherein causing the at least one dormant function of the automated assistant to be activated is further contingent on determining that the user profile is authorized for the client device. ### Claim 8 A method implemented by one or more processors of a client device that facilitates touch-free interaction between one or more users and an automated assistant, the method comprising: processing image frames captured by a camera of the client device; determining, based on processing the image frames: that a gaze of a user is directed toward the client device, and a user profile for the user; processing audio data captured by one or more microphones of the client device; determining, based on processing the audio data, that a spoken utterance, included in the audio data: temporally corresponds to the gaze, and has voice characteristics that match the user profile that is determined based on processing the image frames; and in response to determining the gaze of the user, and contingent on determining that the spoken utterance temporally corresponds to the gaze and has the voice characteristics that match the user profile that is determined based on processing the image frames: causing at least one dormant function of the automated assistant to be activated. ### Claim 9 (depends on 8) The method of claim 8, wherein the at least one dormant function of the automated assistant, that is caused to be activated in response to determining the gaze of the user, and contingent on determining that the spoken utterance temporally corresponds to the gaze and has the voice characteristics that match the user profile that is determined based on processing the image frames comprises: transmitting of data, from the client device, to a remote server associated with the automated assistant. ### Claim 10 (depends on 9) The method of claim 9, wherein the at least one dormant function of the automated assistant, that is caused to be activated in response to determining the gaze of the user, and contingent on determining that the spoken utterance temporally corresponds to the gaze and has the voice characteristics that match the user profile that is determined based on processing the image frames comprises: automatic speech processing of the audio data. ### Claim 11 (depends on 8) The method of claim 8, wherein the at least one dormant function of the automated assistant, that is caused to be activated in response to determining the gaze of the user, and contingent on determining that the spoken utterance temporally corresponds to the gaze and has the voice characteristics that match the user profile that is determined based on processing the image frames further comprises: graphically rendering content that is tailored to the user profile. ### Claim 12 (depends on 8) The method of claim 8, wherein the at least one dormant function of the automated assistant, that is caused to be activated in response to determining the gaze of the user, and contingent on determining that the spoken utterance temporally corresponds to the gaze and has the voice characteristics that match the user profile that is determined based on processing the image frames comprises: automatic speech processing of the audio data. ### Claim 13 (depends on 8) The method of claim 8, wherein determining, based on processing the image frames, the user profile of the user comprises performing facial recognition based on processing at least one of the image frames. ### Claim 14 (depends on 13) The method of claim 13, wherein determining, based on processing the image frames, that the gaze of the user is directed toward the client device comprises processing the image frames using a trained gaze machine learning model stored locally at the client device. ### Claim 15 (depends on 8) The method of claim 8, further comprising: determining that the user profile is authorized for the client device; wherein causing the at least one dormant function of the automated assistant to be activated is further contingent on determining that the user profile is authorized for the client device. ### Claim 16 A client device, comprising: a vision component; a presence sensor; one or more processors, wherein one or more of the processors are configured to: detect, based on a signal from the presence sensor, that a human is present in an environment of the presence sensor; in response to detecting that the human is present in the environment: activate the vision component to provide a stream of vision data that is based on output from the vision component; process the vision data using at least one trained machine learning model stored locally on the client device to monitor for occurrence of both: an invocation gesture of a user captured by the vision data, and a gaze of the user that is directed toward the client device; detect, based on the monitoring, occurrence of both: the invocation gesture, and the gaze; and in response to detecting the occurrence of both the invocation gesture and the gaze: cause at least one dormant function of an automated assistant to be activated. ### Claim 17 (depends on 16) The client device of claim 16, wherein the at least one dormant function of the automated assistant, that is caused to be activated in response to detecting the occurrence of both the invocation gesture and the gaze comprises: transmitting of data, from the client device, to a remote server associated with the automated assistant. ### Claim 18 (depends on 16) The client device of claim 16, wherein the at least one dormant function of the automated assistant, that is caused to be activated in response to detecting the occurrence of both the invocation gesture and the gaze comprises: automatic speech processing of the audio data.