# US 2025/0085774 A1 · Gaze assisted input for an electronic device > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2025/0085774 A1 - Document ID: US20250085774A1 - Status: Published application - Kind: A1 - Application number: 18/464,007 - Filed: 2023-09-08 - Published: 2025-03-13 - Earliest claimed: 2023-09-08 - Assignees: Huawei Technologies Co., Ltd. - Inventors: Roeland Petrus Hubertus VERTEGAAL; Sean Brian BRALEY - Web page: https://patentgrep.com/patent/US20250085774A1 ## CPC - G06F 3/013: {Eye tracking input arrangements (G06F3/015 takes precedence)} - G06F 3/167: {Audio in a user interface, e.g. using voice commands for navigating, audio feedback} - G10L 15/1815: {Semantic context, e.g. disambiguation of the recognition hypotheses based on word meaning} - G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue - G10L 15/30: Distributed recognition, e.g. in client-server systems, for mobile phones or network applications - G06F 2203/0381: Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer - G10L 2015/223: {Execution procedure of a spoken command} - G10L 2015/228: {of application context} - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING ## Abstract A computer implemented method and system for gaze-assisted input that includes displaying a plurality of display elements in a display space, tracking a user's point of gaze within the display space, receiving a speech input, identifying one of the plurality of display elements as a subject display element for the speech input based on the tracking, and automatically performing an action based on the subject display element and the speech input. ## Claims ### Claim 1 A computer implemented method comprising: displaying a plurality of display elements in a display space; tracking a user's point of gaze within the display space by determining coordinates for the user's point of gaze within a coordinate system of the display space; determining a gaze region within the display space based on the determined coordinates for the user's point of gaze, wherein the gaze region encompasses the user's point of gaze and a surrounding region of the display space; receiving a speech input; identifying one of the plurality of display elements as a subject display element for the speech input based on a predetermined minimum threshold spatial overlap between the gaze region and a sub-region of the display space occupied by the subject display element for a predetermined minimum threshold time of the speech input; and automatically performing an action based on the subject display element and the speech input. ### Claim 2 (canceled) ### Claim 3 (canceled) ### Claim 4 (canceled) ### Claim 5 (depends on 1) The method of claim 1 wherein the method is performed by a first electronic device and the subject display element is a GUI element that represents a further device that is associated with the first electronic device, the method comprising: extracting a command from the speech input, and wherein automatically performing the action comprises transmitting the command, using a network interface of the first electronic device, for processing by the further device. ### Claim 6 (depends on 5) The method of claim 5, further comprising causing the GUI element to move in the display space in response to motion of one or both of the further device or the first electronic device. ### Claim 7 (depends on 1) The method of claim 1 wherein the subject display element corresponds to an application, the method comprising: extracting text from the speech input, and wherein automatically performing the action comprises providing the extracted text for processing by the application. ### Claim 8 (depends on 1) The method of claim 1 wherein the subject display element corresponds to a remotely hosted service, the method comprising: extracting a text search query or text prompt from the speech input, and wherein automatically performing the action comprises automatically providing the extracted text search query or text prompt via a communication network to the remotely hosted service and receiving a corresponding response from the remotely hosted service. ### Claim 9 (depends on 1) The method of claim 1 wherein the subject display element corresponds to a word displayed in a text display field, the method comprising: extracting a replacement word from the speech input, and wherein automatically performing the action comprises replacing the word displayed in the text display field with the replacement word or phrase. ### Claim 10 (depends on 1) The method of claim 1 wherein the subject display element includes indicia indicating a meaning, the method comprising: extracting a command from the speech input, and wherein automatically performing the action comprises comparing the extracted command to the meaning indicated by the indicia and causing an activity corresponding to the extracted command to be performed only when the extracted command matches the meaning indicated by the indicia. ### Claim 11 A computing system comprising: one or more processors; and one or more memories storing instructions that, when executed by the one or more processors, configure the computing system to: display a plurality of display elements in a display space; track a user's point of gaze within the display space by determining coordinates for the user's point of gaze within a coordinate system of the display space; determine a gaze region within the display space based on the determined coordinates for the user's point of gaze, wherein the gaze region encompasses the user's point of gaze and a surrounding region of the display space; receive a speech input; identify one of the plurality of display elements as a subject display element for the speech input based on a predetermined minimum threshold spatial overlap between the gaze region and a sub-region of the display space occupied by the subject display element for a predetermined minimum threshold time of the speech input; and automatically perform an action based on the subject display element and the speech input. ### Claim 12 (canceled) ### Claim 13 (canceled) ### Claim 14 (canceled) ### Claim 15 (depends on 11) The computing system of claim 11 wherein the subject display element is a GUI element that represents a further device that is associated with the computing system, the computing system being configured to: extract a command from the speech input, and wherein automatically performing the action comprises transmitting the command, using a network interface of the computing system, for processing by the further device. ### Claim 16 (depends on 11) The computing system of claim 11 wherein the subject display element corresponds to an application, the computing system being configured to: extract text from the speech input, and wherein automatically performing the action comprises providing the extracted text for processing by the application. ### Claim 17 (depends on 11) The computing system of claim 11 wherein the subject display element corresponds to a remotely hosted service, the computing system being configured to: extract a text search query or text prompt from the speech input, and wherein automatically performing the action comprises automatically providing the extracted text search query or text prompt via a communication network to the remotely hosted service and receiving a corresponding response from the remotely hosted service. ### Claim 18 (depends on 11) The computing system of claim 11 wherein the subject display element corresponds to a word displayed in a text display field, the computing system being configured to: extract a replacement word from the speech input, and wherein automatically performing the action comprises replacing the word displayed in the text display field with the replacement word or phrase. ### Claim 19 (depends on 11) The computing system of claim 11 wherein the subject display element includes indicia indicating a meaning, the computing system being configured to: extract a command from the speech input, and wherein automatically performing the action comprises comparing the extracted command to the meaning indicated by the indicia and causing an activity corresponding to the extracted command to be performed only when the extracted command matches the meaning indicated by the indicia. ### Claim 20 A non-transitory computer-readable storage medium, the computer-readable storage medium including instructions that when executed by a computing system, cause the computing system to: display a plurality of display elements in a display space; track a user's point of gaze within the display space by determining coordinates for the user's point of gaze within a coordinate system of the display space; determine a gaze region within the display space based on the determined coordinates for the user's point of gaze, wherein the gaze region encompasses the user's point of gaze and a surrounding region of the display space; receive a speech input; identify one of the plurality of display elements as a subject display element for the speech input based on a predetermined minimum threshold spatial overlap between the gaze region and a sub-region of the display space occupied by the subject display element for a predetermined minimum threshold time of the speech input; and automatically perform an action based on the subject display element and the speech input. ### Claim 21 (depends on 1) The method of claim 1, wherein the predetermined minimum threshold spatial overlap represents at least a percentage of overlap between the gaze region and the sub-region of the display space occupied by the subject display element for at least a percentage of the time duration of the speech input. ### Claim 22 (depends on 1) The method of claim 1, wherein the predetermined minimum threshold spatial overlap represents at least 70% of overlap between the gaze region and the sub-region of the display space occupied by the subject display element for at least 50% of the time duration of the speech input. ### Claim 23 (depends on 1) The method of claim 1, further comprising identifying the one of the plurality of display elements as the subject display element based on an extracted content of the speech input. ### Claim 24 (depends on 11) The computing system of claim 11, wherein the predetermined minimum threshold spatial overlap represents at least a percentage of overlap between the gaze region and the sub-region of the display space occupied by the subject display element for at least a percentage of the time duration of the speech input. ### Claim 25 (depends on 11) The computing system of claim 11, wherein the predetermined minimum threshold spatial overlap represents at least 70% of overlap between the gaze region and the sub-region of the display space occupied by the subject display element for at least 50% of the time duration of the speech input. ### Claim 26 (depends on 11) The computing system of claim 11, the computing system being configured to: identify the one of the plurality of display elements as the subject display element based on an extracted content of the speech input.