# US 2019/0317606 A1 · Multiple User Interaction with Audio Devices Using Speech and Gestures > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2019/0317606 A1 - Document ID: US20190317606A1 - Status: Published application - Kind: A1 - Application number: 15/951,573 - Filed: 2018-04-12 - Published: 2019-10-17 - Earliest claimed: 2018-04-12 - Assignees: International Business Machines Corporation - Applicants: International Business Machines Corporation - Inventors: Mohit Jain; Pratyush Kumar; Anup Agarwal - Web page: https://patentgrep.com/patent/US20190317606A1 ## CPC - G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)} - G06F 3/043: using propagating acoustic waves - G06F 3/167: {Audio in a user interface, e.g. using voice commands for navigating, audio feedback} - G06T 7/262: using transform domain methods, e.g. Fourier domain methods - H03D 1/00: Demodulation of amplitude-modulated oscillations (H03D5/00, H03D9/00, H03D11/00 take precedence) - G06F 2218/16: by matching signal segments - G06F 2218/22: Source localisation; Inverse modelling - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - H03D: DEMODULATION OR TRANSFERENCE OF MODULATION FROM ONE CARRIER TO ANOTHER (masers, lasers H01S; circuits capable of acting both as modulator and demodulator H03C; details applicable to both modulators and frequency-changers H03C; demodulating pulses H03K9/00; transforming types of pulse modulation H03K11/00; coding, decoding or code conversion, in general H03M; repeater stations H04B7/14; demodulators adapted for AC systems of digital information transmission H04L27/00; synchronous demodulators adapted for colour television H04N9/66) ## Abstract Methods, systems and computer program products for speech and gesture interaction of multiple users with devices using voice activated interfaces are provided herein. A computer-implemented method includes transmitting a pilot signal from a device comprising a voice activated interface, the pilot signal having a frequency, detecting motion of at least one user around the device, associating the detected motion with a gesture, and performing a function in response to the gesture. Detecting the motion includes receiving a reflected signal of the pilot signal caused by the motion of the at least one user, and detecting a shift in the reflected signal from the pilot signal, wherein detecting the shift comprises determining one or more differences between a waveform corresponding to the pilot signal and a waveform corresponding to the reflected signal. ## Claims ### Claim 1 A computer-implemented method, comprising: transmitting a pilot signal from a device comprising a voice activated interface, the pilot signal having a frequency; detecting motion of at least one user around the device, wherein said detecting the motion comprises: receiving a reflected signal of the pilot signal caused by the motion of the at least one user; and detecting a shift in the reflected signal from the pilot signal, wherein said detecting the shift comprises determining one or more differences between a waveform corresponding to the pilot signal and a waveform corresponding to the reflected signal, wherein said detecting the shift comprises (i) passing the reflected signal through a high pass filter, and (ii) processing output from the high pass filter utilizing a non-equispaced fast Fourier transform (NFFT); associating the detected motion with a gesture via application of at least one machine learning classification technique, comprising one or more support vector machines, to information pertaining to the detected shift, thereby identifying a shift-related signature caused by the detected motion and mapping the shift-related signature to training data associated with multiple user motions; and performing a function in response to the gesture. ### Claim 2 (depends on 1) The computer-implemented method of claim 1, comprising: mapping a plurality of users to respective locations around the device. ### Claim 3 (depends on 2) The computer-implemented method of claim 2, wherein said mapping comprises determining that the initial locations of two or more of the plurality of users around the device will prevent distinguishing respective motions of the two or more users. ### Claim 4 (depends on 3) The computer-implemented method of claim 3, comprising providing respective audio cues to the two or more users to change their initial locations to locations which allow the respective motions of the two or more users to be distinguished. ### Claim 5 (depends on 3) The computer-implemented method of claim 3, comprising changing the frequency of the pilot signal to a frequency which allows the respective motions of the two or more users to be distinguished. ### Claim 6 (depends on 2) The computer-implemented method of claim 2, comprising: detecting respective motions of the plurality of users in the area of the device; associating the detected respective motions with a plurality of respective gestures; and performing a plurality of respective functions corresponding to operation of the device in response to the plurality of respective gestures. ### Claim 7 (depends on 1) The computer-implemented method of claim 1, wherein said detecting the shift comprises determining a frequency shift of the reflected signal from the frequency of the pilot signal by determining a peak difference between said waveform corresponding to the pilot signal and said waveform corresponding to the reflected signal. ### Claim 8 (canceled) ### Claim 9 (depends on 1) The computer-implemented method of claim 1, wherein said detecting the shift comprises determining a phase shift of the reflected signal by (i) multiplying the reflected signal with a carrier and a phase shifted carrier; and (ii) passing the respective results of said multiplying through low pass filters to obtain I and Q waveforms, respectively. ### Claim 10 (depends on 1) The computer-implemented method of claim 1, comprising: spatially filtering at least one of the pilot signal and the reflected signal with respect to a plurality of microphones in an array in the device. ### Claim 11 (depends on 10) The computer-implemented method of claim 10, wherein said spatially filtering increases the distance from the device over which the motion of the at least one user is detected. ### Claim 12 (depends on 1) The computer-implemented method of claim 1, wherein said associating the detected motion with the gesture comprises matching the detected shift with a corresponding gesture. ### Claim 13 (depends on 1) The computer-implemented method of claim 1, wherein the function comprises one of (i) adjusting audio volume of the device and (ii) navigating through an audio clip being played by the device. ### Claim 14 (depends on 1) The computer implemented method of claim 1, wherein the at least one user is positioned greater than about zero meters to about five meters from the device. ### Claim 15 A computer program product, the computer program product comprising a tangible computer readable storage medium having program instructions embodied therewith, the program instructions executable by a device to cause the device to: transmit a pilot signal from the device, the pilot signal having a frequency and the device comprising a voice activated interface; detect motion of at least one user around the device, wherein in detecting the motion, the program instructions further cause the device to: receive a reflected signal of the pilot signal caused by the motion of the at least one user; and detect a shift in the reflected signal from the pilot signal, wherein in detecting the shift, the program instructions further cause the device to determine one or more differences between a waveform corresponding to the pilot signal and a waveform corresponding to the reflected signal, wherein said detecting the shift comprises (i) passing the reflected signal through a high pass filter, and (ii) processing output from the high pass filter utilizing a non-equispaced fast Fourier transform (NFFT); associate the detected motion with a gesture via application of at least one machine learning classification technique, comprising one or more support vector machines, to information pertaining to the detected shift, thereby identifying a shift-related signature caused by the detected motion and mapping the shift-related signature to training data associated with multiple user motions; and perform a function in response to the gesture. ### Claim 16 A system comprising: a memory; and at least one processor coupled to the memory and configured for: transmitting a pilot signal from a device comprising a voice activated interface, the pilot signal having a frequency; detecting motion of at least one user around the device, wherein said detecting the motion comprises: receiving a reflected signal of the pilot signal caused by the motion of the at least one user; and detecting a shift in the reflected signal from the pilot signal, wherein said detecting the shift comprises determining one or more differences between a waveform corresponding to the pilot signal and a waveform corresponding to the reflected signal, wherein said detecting the shift comprises (i) passing the reflected signal through a high pass filter, and (ii) processing output from the high pass filter utilizing a non-equispaced fast Fourier transform (NFFT); associating the detected motion with a gesture via application of at least one machine learning classification technique, comprising one or more support vector machines, to information pertaining to the detected shift, thereby identifying a shift-related signature caused by the detected motion and mapping the shift-related signature to training data associated with multiple user motions; and performing a function in response to the gesture. ### Claim 17 (depends on 16) The system according to claim 16, wherein the at least one processor is further configured for mapping a plurality of users to respective locations around the device. ### Claim 18 A computer-implemented method, comprising: transmitting a pilot signal from a device comprising a voice activated interface, the pilot signal having a frequency; mapping a plurality of users to respective locations around the device; detecting respective motions of the plurality of users around the device, wherein said detecting the motion comprises: receiving respective reflected signals of the pilot signal from each of the plurality of users caused by the respective motions of the plurality of users; and detecting respective shifts in the respective reflected signals from the pilot signal, wherein said detecting the respective shifts comprises determining one or more differences between a waveform corresponding to the pilot signal and waveforms corresponding to the respective reflected signals, wherein said detecting the shifts comprises (i) passing the reflected signals through a high pass filter, and (ii) processing output from the high pass filter utilizing a non-equispaced fast Fourier transform (NFFT); associating the detected respective motions with respective gestures via application of at least one machine learning classification technique, comprising one or more support vector machines, to information pertaining to the detected shifts, thereby identifying shift-related signatures caused by the detected motions and mapping the shift-related signatures to training data associated with multiple user motions; and performing respective functions in response to the respective gestures. ### Claim 19 (depends on 18) The computer-implemented method of claim 18, wherein said mapping comprises determining that the initial locations of the plurality of users around the device will prevent distinguishing the respective motions of the plurality of users. ### Claim 20 (depends on 19) The computer-implemented method of claim 19, comprising at least one of: providing respective audio cues to the plurality of users to change their initial locations to locations which allow the respective motions of the plurality of users to be distinguished; and changing the frequency of the pilot signal to a frequency which allows the respective motions of the plurality of users to be distinguished.