# US 10,481,696 B2 · Radar based user interface > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,481,696 B2 - Document ID: US10481696B2 - Status: Granted - Kind: B2 - Application number: 15/060,545 - Filed: 2016-03-03 - Published: 2016-09-08 - Granted: 2019-11-19 - Earliest claimed: 2015-03-03 - Assignees: NVIDIA Corporation - Applicants: NVIDIA Corporation - Inventors: Pavlo Molchanov; Shalini Gupta; Kihwan Kim; Kari Pulli - Web page: https://patentgrep.com/patent/US10481696B2 ## CPC - G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)} - B60R 11/04: Mounting of cameras operative during drive; Arrangement of controls thereof relative to the vehicle - G01S 7/0233: {Avoidance by phase multiplex} - G01S 7/0234: {Avoidance by code multiplex} - G01S 7/352: {Receivers} - G01S 7/415: {Identification of targets based on measurements of movement associated with the target} - G01S 13/42: Simultaneous measurement of distance and other co-ordinates (indirect measurement G01S13/46) - G01S 13/584: {adapted for simultaneous range and velocity measurements} - G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)} - G06F 18/214: Generating training patterns; Bootstrap methods, e.g. bagging or boosting - G06F 18/24: Classification techniques - G06F 18/25: Fusion techniques - G06N 3/045: Combinations of networks - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/082: modifying the architecture, e.g. adding, deleting or silencing nodes or connections - G06N 3/084: Backpropagation, e.g. using gradient descent - G06N 3/09: Supervised learning - G06V 40/28: {Recognition of hand or arm movements, e.g. recognition of deaf sign language (static hand signs G06V40/113)} - G01S 13/4454: {phase comparisons monopulse, i.e. comparing the echo signals received by an interferometric antenna arrangement} - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - B60R: VEHICLES, VEHICLE FITTINGS, OR VEHICLE PARTS, NOT OTHERWISE PROVIDED FOR - G01S: RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract An apparatus and method for radar based gesture detection. The apparatus includes a processing element and a transmitter configured to transmit radar signals. The transmitter is coupled to the processing element. The apparatus further includes a plurality of receivers configured to receive radar signal reflections, where the plurality of receivers is coupled to the processing element. The transmitter and plurality of receivers are configured for short range radar and the processing element is configured to detect a hand gesture based on the radar signal reflections received by the plurality of receivers. ## Claims ### Claim 1 An apparatus for gesture sensing comprising: a processing element; a depth sensor communicatively coupled to the processing element; a color sensor; a transmitter configured to transmit radar signals, wherein the transmitter is coupled to the processing element; a plurality of receivers configured to receive radar signal reflections, wherein the plurality of receivers is coupled to the processing element, and the processing element is configured to: cross-calibrate the color sensor, the plurality of receivers, and the depth sensor; generate location and velocity data based on the radar signal reflections received by the plurality of receivers and by performing a transform over time process; register information received from the plurality of receivers with the information received from the depth sensor based on a cross-calibration result; and determine a hand gesture based on the location and velocity data; wherein the transmitter and plurality of receivers are configured for short range radar based detection, and wherein the location and velocity data are generated further based on information generated by the depth sensor and information generated by the color sensor. ### Claim 2 (depends on 1) The apparatus as described in claim 1, wherein the short range radar based detection has a range of one meter. ### Claim 3 (depends on 1) The apparatus as described in claim 1, wherein the transmitter and plurality of receivers are a portion of a user interface device of a vehicle. ### Claim 4 (depends on 1) The apparatus as described in claim 1, wherein a first pair of receivers of the plurality of receivers is configured to measure a vertical component of a gesture movement and a second pair of receivers of the plurality of receivers is configured to measure a horizontal component of a gesture movement. ### Claim 5 (depends on 1) The apparatus as described in claim 1, wherein the processing element is configured to determine a 3D position and radial velocity for a signal reflection received by the plurality of receivers. ### Claim 6 (depends on 5) The apparatus as described in claim 5, wherein the processing element is further configured to determine an elevation angle and an azimuth angle for an object from which a signal reflection received by the plurality of receivers. ### Claim 7 (depends on 1) The apparatus as described in claim 1, wherein the transmitter and the plurality of receivers are operated in a frequency-modulated continuous-wave (FMCW) mode. ### Claim 8 (depends on 1) The apparatus as described in claim 1, wherein the transmitter and the plurality of receivers are part of a multiple input single output (MISO) device. ### Claim 9 (depends on 1) The apparatus as described in claim 1, wherein the hand gesture is a dynamic hand gesture and the processing element is configured to automatically recognize the hand gesture and to determine a command associated with the dynamic hand gesture. ### Claim 10 (depends on 1) The apparatus as described in claim 1, wherein the plurality of receivers is spatially distributed. ### Claim 11 (depends on 1) The apparatus as described in claim 1, wherein the transform over time process comprises a Fast Fourier Transform process. ### Claim 12 (depends on 1) The apparatus as described in claim 1 further comprising a microcontroller separate from the processing element and configured to perform static background subtraction. ### Claim 13 (depends on 1) The apparatus as described in claim 1, wherein the determining a hand gesture based on the location and velocity data comprises a pattern recognition process by using a deep neural network. ### Claim 14 (depends on 1) The apparatus as described in claim 1, wherein the transform over time process comprises a wavelet analysis process. ### Claim 15 (depends on 1) The apparatus as described in claim 1, wherein the transform over time process comprises a super resolution process. ### Claim 16 A system for hand gesture detection and classification, the system comprising: a processing element; a depth sensor communicatively coupled to the processing element; a color sensor; a transmitter configured for transmitting radar signals, wherein the transmitter is coupled to the processing element; a plurality receivers comprising a first receiver, a second receiver, and a third receiver and configured for receiving the radar signals, and wherein the first receiver and the second receiver are configured for measuring a horizontal component of a movement and the second receiver and the third receiver are configured for measuring a vertical component of a movement, and wherein the plurality of receivers are communicatively coupled to the processing element; and the processing element is configured to: cross-calibrate the color sensor, the depth sensor, and the plurality of receivers; register information received from the plurality of receivers with information received from the depth sensor based on a cross-calibration result; generate location and velocity data based on reflections of the radar signals received by the plurality of receivers and based on information generated by the depth sensor and by performing a transform over time process; and determine a hand gesture based on the location and velocity data; wherein the transmitter and the plurality of receivers are configured for short range radar based detection, and wherein the location and velocity data are generated further based on information generated by the color sensor. ### Claim 17 (depends on 16) The system as described in claim 16, wherein the short range based detection of the hand gestures has a range of one meter. ### Claim 18 (depends on 16) The system as described in claim 16, wherein the transmitter and the plurality of receivers are a portion of a user interface device of a vehicle. ### Claim 19 (depends on 16) The system as described in claim 16, wherein the processing element is configured to determine a 3D position and radial velocity for a signal reflection received from the plurality of receivers. ### Claim 20 (depends on 19) The system as described in claim 19, wherein the processing element is further configured to determine an elevation angle and an azimuth angle for the signal reflection received from the plurality of receivers. ### Claim 21 (depends on 16) The system as described in claim 16, wherein the transmitter and the plurality of receivers are operated in a frequency-modulated continuous-wave (FMCW) mode. ### Claim 22 (depends on 16) The system as described in claim 16, wherein the transform over time process comprises a Fast Fourier Transform process. ### Claim 23 (depends on 16) The system as described in claim 16, wherein determining a hand gesture based on the location and velocity data comprises a pattern recognition process by using a deep neural network. ### Claim 24 (depends on 16) The system as described in claim 16, wherein the transform over time process comprises a wavelet analysis process. ### Claim 25 (depends on 4) The system as described in claim 4, wherein the transform over time process comprises a super resolution process. ### Claim 26 A mobile apparatus comprising: a processing element; a depth sensor communicatively coupled to the processing element; a color sensor; a transmitter configured for transmitting radar signals, wherein the transmitter is coupled to the processing element; a plurality of receivers comprising a first receiver, a second receiver, and a third receiver and configured for receiving the radar signals, and wherein the first receiver and the second receiver are configured for measuring a horizontal component of a movement and the second receiver and the third receiver are configured for measuring a vertical component of a movement, and wherein the a plurality of receivers are communicatively coupled to the processing element; and the processing element is configured to: cross-calibrate the color sensor, the depth sensor, and the plurality of receivers; generate location and velocity data based on reflections of the radar signals received by the plurality of receivers and by performing a transform over time process; detect a hand gesture of a driver of the mobile apparatus based on the location and velocity data; and automatically recognize the hand gesture and to determine a command associated with the hand gesture; wherein the transmitter and the a plurality of receivers are configured for short range radar based detection, and wherein the location and velocity data are generated further based on information generated by the color sensor and based on information generated by the depth sensor. ### Claim 27 (depends on 26) The mobile apparatus as described in claim 26, wherein the short range based detection has a range of substantially one meter. ### Claim 28 (depends on 26) The mobile apparatus as described in claim 26, wherein the processing element is further configured to perform background subtraction. ### Claim 29 (depends on 26) The mobile apparatus as described in claim 26, wherein the processing element is configured to determine a 3D position and radial velocity for a signal reflection from an object received from the plurality of receivers. ### Claim 30 (depends on 26) The mobile apparatus as described in claim 26, wherein the transform over time process comprises a Fast Fourier Transform process. ### Claim 31 (depends on 26) The mobile apparatus as described in claim 26, wherein the transform over time process comprises a wavelet analysis process. ### Claim 32 (depends on 26) The mobile apparatus as described in claim 26, wherein the transform over time process comprises a super resolution process.