# US 12,111,976 B2 · Gesture detection using external sensors > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,111,976 B2 - Document ID: US12111976B2 - Status: Granted - Kind: B2 - Application number: 18/491,837 - Filed: 2023-10-23 - Published: 2024-05-09 - Granted: 2024-10-08 - Earliest claimed: 2019-04-03 - Assignees: Google Inc. - Applicants: Google LLC - Inventors: Katherine Blair Huffman; Gregory Granito - Web page: https://patentgrep.com/patent/US12111976B2 ## CPC - G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)} - G01S 7/415: {Identification of targets based on measurements of movement associated with the target} - G01S 13/867: {Combination of radar systems with cameras} - G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)} - G06F 3/0346: with detection of the device orientation or free movement in a three-dimensional [3D] space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors - G06F 3/167: {Audio in a user interface, e.g. using voice commands for navigating, audio feedback} - G06T 7/20: Analysis of motion (motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51) - G06V 40/28: {Recognition of hand or arm movements, e.g. recognition of deaf sign language (static hand signs G06V40/113)} - G06T 2207/30196: Human being; Person - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G01S: RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract The technology provides for a system for determining a gesture provided by a user. In this regard, one or more processors of the system may receive image data from one or more visual sensors of the system capturing a motion of the user, and may receive motion data from one or more wearable computing devices worn by the user. The one or more processors may recognize, based on the image data, a portion of the user's body that corresponds to a gesture to perform a command. The one or more processors may also determine one or more correlations between the image data and the received motion data. Based on the recognized portion of the user's body and the one or more correlations between the image data and the received motion data, the one or more processors may detect the gesture. ## Claims ### Claim 1 A method, comprising: comparing, by one or more processors, first audio data received from a first audio sensor of a first computing device with second audio data received from a second audio sensor of a second computing device; detecting, by the one or more processors, a gesture made by a portion of a body of a user based on a result of the comparing; determining, by the one or more processors, whether the gesture corresponds to a user command; and controlling, by the one or more processors, one or more functions of the first computing device based on the user command. ### Claim 2 (depends on 1) The method of claim 1, further comprising: obtaining image data from one or more visual sensors; and determining, by the one or more processors, a set of correlations between the image data and at least one of the first audio data or the second audio data. ### Claim 3 (depends on 2) The method of claim 2, wherein determining whether the gesture corresponds to the user command is further based on the set of correlations. ### Claim 4 (depends on 1) The method of claim 1, wherein the user command comprises a voice command generated as the gesture is made. ### Claim 5 (depends on 1) The method of claim 1, wherein the first audio data includes information regarding at least one of a volume or a frequency of a voice command. ### Claim 6 (depends on 1) The method of claim 1, wherein the first audio data is associated with one or more timestamps provided by a first clock of the first computing device. ### Claim 7 (depends on 6) The method of claim 6, wherein the second audio data is associated with one or more timestamps provided by a second clock of the second computing device. ### Claim 8 (depends on 1) The method of claim 1, further comprising, when the first audio data decreases in volume between a first time and a second time, while the second audio data remains unchanged between the first time and the second time, determining that the portion of the body used to make the gesture has moved closer to the second computing device between the first time and the second time. ### Claim 9 (depends on 1) The method of claim 1, further comprising, when the first audio data increases in volume by a predetermined amount between a first time and a second time, while the second audio data increases in volume by an amount less that the predetermined amount between the first time and the second time, determining that the portion of the body used to make the gesture has moved away from the second computing device between the first time and the second time. ### Claim 10 A method, comprising: obtaining, by one or more processors, image data from one or more visual sensors of a first computing device; obtaining, by the one or more processors, audio data from one or more audio sensors of a second computing device; determining, by the one or more processors, a set of correlations between the image data and the audio data; and detecting, by the one or more processors using the set of correlations, a user gesture for performance of a command to operate either the first computing device or the second computing device. ### Claim 11 (depends on 10) The method of claim 10, further comprising: controlling, by the one or more processors based on the detecting, at least one of the first computing device or the second computing device. ### Claim 12 (depends on 10) The method of claim 10, further comprising: receiving radar measurements from one or more radar sensors, wherein detecting the user gesture is based on the received radar measurements. ### Claim 13 A system, comprising: a first computing device including a first audio sensor; a second computing device including a second audio sensor; and one or more processors configured to: compare first audio data received from the first audio sensor with second audio data received from the second audio sensor to obtain a result; detect a gesture made by a portion of a body of a user based on the result; determine whether the gesture corresponds to a user command; and control one or more functions of the first computing device based on the user command. ### Claim 14 (depends on 13) The system of claim 13, wherein the one or more processors are further configured to: obtain image data from one or more visual sensors; and determine a set of correlations between the image data and at least one of the first audio data or the second audio data. ### Claim 15 (depends on 13) The system of claim 13, wherein the first audio data is associated with one or more timestamps provided by a first clock of the first computing device. ### Claim 16 (depends on 15) The system of claim 15, wherein the second audio data is associated with one or more timestamps provided by a second clock of the second computing device. ### Claim 17 A system, comprising: a first computing device including one or more visual sensors; a second computing device including one or more audio sensors; and one or more processors configured to: obtain image data from the one or more visual sensors; obtain audio data from the one or more audio sensors; determine a set of correlations between the image data and the audio data; and detect, using the set of correlations, a user gesture for performance of a command to operate either the first computing device or the second computing device. ### Claim 18 (depends on 17) The system of claim 17, further comprising: one or more radar sensors, wherein the one or more processors are further configured to receive radar measurements from the one or more radar sensors, and the user gesture is detected based on the received radar measurements. ### Claim 19 (depends on 17) The system of claim 17, further comprising: a memory configured to store a set of commands, wherein the one or more processors are further configured to determine whether the user gesture corresponds to one of the set of commands stored in the memory. ### Claim 20 (depends on 17) The system of claim 17, wherein the one or more processors are further configured to control, based on the detecting, at least one of the first computing device or the second computing device. ### Claim 21 (depends on 17) The system of claim 17, wherein at least one of the first computing device or the second computing device is a wearable computing device. ### Claim 22 (depends on 21) The system of claim 21, wherein the one or more visual sensors are configured to capture motion of a user. ### Claim 23 (depends on 17) The system of claim 17, wherein the one or more audio sensors are configured to detect a voice command from a user. ### Claim 24 (depends on 17) The system of claim 17, wherein the set of correlations between the image data and the audio data is determined by synchronizing timestamps of the audio data with the image data. ### Claim 25 (depends on 17) The system of claim 17, wherein the set of correlations between the image data and the audio data is determined by transforming information that is provided in a coordinate system of the first computing device and the second computing device to a coordinate system of the image data. ### Claim 26 (depends on 17) The system of claim 17, wherein the set of correlations between the image data and the audio data is determined by determining relative positions of the first computing device and the second computing device. ### Claim 27 A non-transitory computer readable medium on which instructions are stored, the instructions, when executed by one or more processors, cause the one or more processors to perform a method, the method comprising: comparing first audio data received from a first audio sensor of a first computing device with second audio data received from a second audio sensor of a second computing device; detecting a gesture made by a portion of a body of a user based on a result of the comparing; determining whether the gesture corresponds to a user command; and controlling one or more functions of the first computing device based on the user command. ### Claim 28 (depends on 27) The non-transitory computer readable medium of claim 27, wherein the one or more processors are further configured to: obtain image data from one or more visual sensors; and determine a set of correlations between the image data and at least one of the first audio data or the second audio data. ### Claim 29 A non-transitory computer readable medium on which instructions are stored, the instructions, when executed by one or more processors, cause the one or more processors to perform a method, the method comprising: obtaining image data from one or more visual sensors of a first computing device; obtaining audio data from one or more audio sensors of a second computing device; determining a set of correlations between the image data and the audio data; and detecting, using the set of correlations, a user gesture for performance of a command to operate either the first computing device or the second computing device. ### Claim 30 (depends on 29) The non-transitory computer readable medium of claim 29, wherein the method further comprises: receiving radar measurements from one or more radar sensors, wherein detecting the user gesture is based on the received radar measurements.