# US 10,768,712 B2 · Gesture component with gesture library > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,768,712 B2 - Document ID: US10768712B2 - Status: Granted - Kind: B2 - Application number: 16/401,611 - Filed: 2019-05-02 - Published: 2019-08-22 - Granted: 2020-09-08 - Earliest claimed: 2015-10-06 - Assignees: Google Inc. - Applicants: Google LLC - Inventors: Carsten C. Schwesig; Ivan Poupyrev - Web page: https://patentgrep.com/patent/US10768712B2 ## CPC - G01S 7/415: {Identification of targets based on measurements of movement associated with the target} - A63F 13/21: characterised by their sensors, purposes or types - A63F 13/24: Constructional details thereof, e.g. game controllers with detachable joystick handles - G01S 7/4004: {of parts of a radar system} - G01S 7/41: using analysis of echo signal for target characterisation; Target signature; Target cross-section - G01S 7/412: {based on a comparison between measured values and known or stored values} - G01S 13/56: for presence detection {(presence detection using near field arrangements G01V3/00, e.g. G01V3/08, G01V3/12; burglar, theft or intruder alarms with electrical actuation G08B13/22 - G08B13/26)} - G01S 13/66: Radar-tracking systems; Analogous systems - G01S 13/86: Combinations of radar systems with non-radar systems, e.g. sonar, direction finder - G01S 13/867: {Combination of radar systems with cameras} - G01S 13/88: Radar or analogous systems specially adapted for specific applications (electromagnetic prospecting or detecting of objects, e.g. near-field detection, G01V3/00) - G01S 13/888: {through wall detection} - G01S 13/90: using synthetic aperture techniques {, e.g. synthetic aperture radar [SAR] techniques} - G01S 13/904: {SAR modes} - G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)} - G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)} - G06F 3/04815: Interaction with a metaphor-based environment or interaction object displayed as three-dimensional [3D], e.g. changing the user viewpoint with respect to the environment or object - G06F 16/245: Query processing - G06F 18/217: {Validation; Performance evaluation; Active pattern learning techniques} - G06F 18/25: Fusion techniques - G06F 18/253: {of extracted features} - G06F 18/28: Determining representative reference patterns, e.g. by averaging or distorting; Generating dictionaries - G06F 18/41: {Interactive pattern learning with a human teacher} - G06F 21/32: using biometric data, e.g. fingerprints, iris scans or voiceprints - G06F 21/6245: {Protecting personal data, e.g. for financial or medical purposes} - G06N 20/00: Machine learning - G06V 10/806: {of extracted features} - G06V 20/64: Three-dimensional [3D] objects - G06V 40/28: {Recognition of hand or arm movements, e.g. recognition of deaf sign language (static hand signs G06V40/113)} - H04Q 9/00: Arrangements in telecontrol or telemetry systems for selectively calling a substation from a main station, in which substation desired apparatus is selected for applying a control signal thereto or for obtaining measured values therefrom - H04W 4/80: Services using short range communication, e.g. near-field communication [NFC], radio-frequency identification [RFID] or low energy communication - H04W 16/28: using beam steering - A63F 2300/8082: Virtual reality - G01S 13/865: {Combination of radar systems with lidar systems} - G01S 13/931: of land vehicles - G01S 19/42: Determining position - G01S 2013/9322: {using additional data, e.g. driver condition, road state or weather data} - G06F 1/163: {Wearable computers, e.g. on a belt} - G06F 3/0346: with detection of the device orientation or free movement in a three-dimensional [3D] space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors - G06F 3/0484: for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range - G06F 3/165: {Management of the audio stream, e.g. setting of volume, audio stream path} - G06F 2203/0384: Wireless input, i.e. hardware and software details of wireless interface arrangements for pointing devices - G06F 2221/2105: Dual mode as a secondary aspect - G06T 7/75: {involving models} - G08C 17/02: using a radio link - G08C 2201/93: Remote control using other portable devices, e.g. mobile phone, PDA, laptop - H04Q 2209/883: where the sensing device enters an active or inactive mode - G01S: RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES - A63F: CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - H04Q: SELECTING (switches, relays, selectors H01H; wireless communication networks H04W) - H04W: WIRELESS COMMUNICATION NETWORKS (broadcast communication H04H; communication systems using wireless links for non-selective communication, e.g. wireless extensions H04M1/72) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G08C: TRANSMISSION SYSTEMS FOR MEASURED VALUES, CONTROL OR SIMILAR SIGNALS (fluid pressure transmitting systems F15B; mechanical means for transferring the output of a sensing member into a different variable G01D5/00; mechanical control systems G05G) ## Abstract A gesture component with a gesture library is described. The gesture component is configured to expose operations for execution by application of a computing device based on detected gestures. In one example, an input is detected using a three dimensional object detection system of a gesture component of the computing device. A gesture is recognized by the gesture component based on the detected input through comparison with a library of gestures maintained by the gesture component. An operation is then recognized that corresponds to the gesture by the gesture component using the library of gestures. The operation is exposed by the gesture component via an application programming interface to at least one application executed by the computing device to control performance of the operation by the at least one application. ## Claims ### Claim 1 A method of controlling operation of a computing device by radar-based gesture detection, the method comprising: receiving, by the computing device, raw data representing radar signals that have been reflected off an object; transforming, by the computing device, the raw data of the radar signals into one or more different representations that represent information about the object, the transforming comprising at least one of: an I/Q transformation that yields a complex vector containing phase and amplitude information about the object, a beamforming transformation that yields spatial information about the object, a range-Doppler transformation that yields range rate and distance information about the object, a range profile transformation that yields recognition information about the object, a micro-Doppler transformation that yields high-resolution recognition information about the object, or a spectrogram transformation that yields a spectrogram of corresponding frequencies of the raw data of the radar signals; extracting, by the computing device, feature information about the object from the one or more different representations that represent information about the object; recognizing, by the computing device, a gesture performed by the object, the recognizing based on a comparison of the extracted feature information about the object with a library of gestures maintained by the computing device; recognizing, by the computing device, an operation that corresponds to the gesture using the library of gestures; and exposing, by the computing device, an indication of the operation for receipt by at least one application executed by the computing device or a remote device to cause the at least one application to control performance of the operation. ### Claim 2 (depends on 1) The method of claim 1, wherein the radar signals comprise radio signals. ### Claim 3 (depends on 2) The method of claim 2, wherein the radar signals correspond to a 60 GHz band. ### Claim 4 (depends on 1) The method of claim 1, wherein the at least one application is not made aware as to how the operation is detected by the computing device. ### Claim 5 (depends on 1) The method of claim 1, wherein the library of gestures includes media gestures usable by the at least one application to control rendering of media. ### Claim 6 (depends on 1) The method of claim 1, wherein the library of gestures includes user interface navigation gestures usable by the at least one application to control navigation within a user interface. ### Claim 7 (depends on 1) The method of claim 1, wherein: the one or more different representations represent a classification of the object such as a hand, a finger, or a head; and the extracting the feature information about the object comprises looking for feature information specific to the classification of the object. ### Claim 8 (depends on 1) The method of claim 1, wherein the transforming further comprises the beamforming transformation. ### Claim 9 (depends on 1) The method of claim 1, wherein the transforming further comprises the range-Doppler transformation. ### Claim 10 (depends on 1) The method of claim 1, wherein the transforming further comprises the range profile transformation. ### Claim 11 (depends on 1) The method of claim 1, wherein the transforming further comprises the micro-Doppler transformation. ### Claim 12 (depends on 1) The method of claim 1, wherein the transforming further comprises the spectrogram transformation. ### Claim 13 (depends on 1) The method of claim 1, wherein the transforming further comprises the spectrogram transformation, the range-Doppler transformation, and the beamforming transformation. ### Claim 14 (depends on 1) The method of claim 1, wherein the transforming further comprises the range profile transformation and the micro-Doppler transformation. ### Claim 15 A computing device comprising: a processing system; an application maintained in computer-readable storage media and executable by the processing system to perform a plurality of operations; and a gesture component implemented at least partially in hardware, the gesture component configured to: receive raw data representing radar signals that have been reflected by an object; transform the raw data of the radar signals into a different representation that represents information about the object, the transform comprising at least one of: an I/Q transformation that yields a complex vector containing phase and amplitude information about the object, a beamforming transformation that yields spatial information about the object, a range-Doppler transformation that yields range rate and distance information about the object, a range profile transformation that yields recognition information about the object, a micro-Doppler transformation that yields high-resolution recognition information about the object, or a spectrogram transformation that yields a spectrogram of corresponding frequencies of the raw data of the radar signals; extract feature information about the object from the different representation that represents information about the object; maintain a library of gestures corresponding to the operations; compare the feature information about the object from the transformed raw data of the radar signals with the library of gestures; recognize a gesture performed by the object and a corresponding operation based on the comparison; and expose an indication of the corresponding operation that corresponds to the recognized gesture for receipt by the application to cause the application to control performance of the corresponding operation. ### Claim 16 (depends on 15) The computing device of claim 15, wherein the radar signals comprise radio waves. ### Claim 17 (depends on 16) The computing device of claim 16, wherein the radio waves correspond to a 60 GHz band. ### Claim 18 (depends on 15) The computing device of claim 15, wherein the application is not made aware as to how the corresponding operation is detected by the gesture component. ### Claim 19 (depends on 15) The computing device of claim 15, wherein: the operations comprise media control operations; and the library of gestures includes media gestures corresponding to the media control operations. ### Claim 20 (depends on 15) The computing device of claim 15, wherein: the operations comprise user interface navigation control operations; and the library of gestures includes user interface navigation gestures corresponding to the user interface navigation control operations. ### Claim 21 (depends on 15) The computing device of claim 15, wherein: the different representation represents a classification of the object such as a hand, a finger, or a head; and the extracting the feature information about the object comprises looking for feature information specific to the classification of the object. ### Claim 22 (depends on 15) The computing device of claim 15, wherein the transform further comprises the I/Q transformation. ### Claim 23 (depends on 15) The computing device of claim 15, wherein the transform further comprises the beamforming transformation. ### Claim 24 (depends on 15) The computing device of claim 15, wherein the transform further comprises the range-Doppler transformation. ### Claim 25 (depends on 15) The computing device of claim 15, wherein the transform further comprises the range profile transformation. ### Claim 26 (depends on 15) The computing device of claim 15, wherein the transform further comprises the micro-Doppler transformation. ### Claim 27 (depends on 15) The computing device of claim 15, wherein the transform further comprises the spectrogram transformation. ### Claim 28 (depends on 15) The computing device of claim 15, wherein the transform further comprises the spectrogram transformation, the range-Doppler transformation, and the beamforming transformation. ### Claim 29 (depends on 15) The computing device of claim 15, wherein the transform further comprises the range profile transformation and the micro-Doppler transformation. ### Claim 30 A method performed by a computing device of exposing an operation for an application from radar-based gesture detection, the method comprising: transmitting, by the computing device, a plurality of outgoing radio frequency radar signals; capturing, by the computing device, incoming radio frequency radar signals generated by the outgoing radio frequency radar signals reflecting off a user; exposing, by the computing device, the operation to the application based on the captured incoming radio frequency radar signals, the operation exposed by: transforming raw data corresponding to the captured incoming radio frequency radar signals into a different representation that represents information about a portion of the user, the transforming comprising at least one of: an I/Q transformation that yields a complex vector containing phase and amplitude information about the object, a beamforming transformation that yields spatial information about the object, a range-Doppler transformation that yields range rate and distance information about the object, a range profile transformation that yields recognition information about the object, a micro-Doppler transformation that yields high-resolution recognition information about the object, or a spectrogram transformation that yields a spectrogram of corresponding frequencies of the raw data of the radar signals; extracting feature information about the portion of the user based on the different representation that represents information about the portion of the user; comparing the feature information about the portion of the user with a library of gestures maintained by the computing device; recognizing a gesture performed by the portion of the user and the operation based on the comparison, the operation corresponding to the recognized gesture; and exposing the operation that corresponds to the recognized gesture to the application such that the application can control performance of the operation. ### Claim 31 (depends on 30) The method of claim 30, wherein the outgoing radio frequency radar signals correspond to a 60 GHz band. ### Claim 32 (depends on 30) The method of claim 30, wherein the operation comprises a media-related operation or a navigation operation. ### Claim 33 (depends on 30) The method of claim 30, wherein: the portion of the user comprises a body part of the user such as a hand, a finger, or a head; and the extracting the feature information comprises searching for feature information specific to the body part of the user.