Using ultrasound to improve IMU-based gesture detection

A non-transitory computer-readable storage medium encoded with a computer program, the program comprising instructions that upon execution by a computer cause the computer to perform operations comprising:

• determining, based on accelerometer or gyroscope data generated by one or more sensors on a first computing device, that a user of the first computing device is likely to have begun performing movement using the first computing device;
• after determining that the user of the first computing device is likely to have begun performing movement using the first computing device and before the first computing device has identified a particular gesture corresponding to the gesture, transmitting, to a different, physically separate, second computing device, data indicating that the user of the first computing device is likely to have begun performing the movement using the first computing device;
• before the first computing device has identified the particular gesture corresponding to the movement, obtaining, by the first computing device, audio data corresponding to a sound generated by the second computing device in response to the first computing device transmitting data to the different, physically separate, second computing device indicating that the user of the first computing device was likely to have begun performing the movement using the first computing device;
• before the first computing device has identified the particular gesture corresponding to the movement, identifying one or more frequency shift peak values from the audio data;
• and
• identifying, by the first computing device, the particular gesture corresponding to the movement based at least on (i) the accelerometer or gyroscope data that was used to determine that the user of the first computing device was likely to have begun performing the movement using the first computing device, and (ii) the one or more of the frequency shift peak values that are identified from the audio data that (I) corresponds to the sound generated by the second computing device and (II) was obtained in response to the first computing device transmitting data to the different, physically separate, second computing device indicating that the user of the first computing device was likely to have begun performing the movement using the first computing device.

A system comprising:

• one or more processors and one or more computer storage media storing instructions that are operable, when executed by the one or more processors, to cause the one or more processors to perform operations comprising:
• determining, based on accelerometer or gyroscope data generated by one or more sensors on a first computing device, that a user of the first computing device is likely to have begun performing movement using the first computing device;
• after determining that the user of the first computing device is likely to have begun performing the movement using the first computing device and before the first computing device has identified a particular gesture corresponding to the movement, transmitting, to a different, physically separate, second computing device, data indicating that the user of the first computing device is likely to have begun performing the movement using the first computing device;
• before the first computing device has identified the particular gesture corresponding to the movement, obtaining, by the first computing device, audio data corresponding to a sound generated by the second computing device in response to the first computing device transmitting data to the different, physically separate, second computing device indicating that the user of the first computing device was likely to have begun performing the movement using the first computing device;
• before the first computing device has identified the particular gesture corresponding to the movement, identifying one or more frequency shift peak values from the audio data;
• and
• identifying, by the first computing device, the particular gesture corresponding to the movement based at least on (i) the accelerometer or gyroscope data that was used to determine that the user of the first computing device was likely to have begun performing the movement using the first computing device, and (ii) the one or more of the frequency shift peak values that are identified from the audio data that (I) corresponds to the sound generated by the second computing device and (II) was obtained in response to the first computing device transmitting data to the different, physically separate, second computing device indicating that the user of the first computing device was likely to have begun performing the movement using the first computing device.