# US 9,436,287 B2 · Systems and methods for switching processing modes using gestures > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 9,436,287 B2 - Document ID: US9436287B2 - Status: Granted - Kind: B2 - Application number: 13/835,234 - Filed: 2013-03-15 - Published: 2014-09-18 - Granted: 2016-09-06 - Earliest claimed: 2013-03-15 - Assignees: QUALCOMM Incorporated - Applicants: QUALCOMM Incorporated - Inventors: Phuong L. Ton; Evan R. Hildreth; Joel S. Bernarte; Suzana Arellano - Web page: https://patentgrep.com/patent/US9436287B2 ## CPC - G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)} - G06F 3/0304: {Detection arrangements using opto-electronic means (constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042)} - G06F 3/167: {Audio in a user interface, e.g. using voice commands for navigating, audio feedback} - G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue - G10L 15/26: Speech to text systems (G10L15/08 takes precedence) - G10L 21/00: Speech or voice signal processing techniques to produce another audible or non-audible signal, e.g. visual or tactile, in order to modify its quality or its intelligibility (G10L19/00 takes precedence) - G06F 3/005: {Input arrangements through a video camera} - G06F 2203/0381: Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer - G10L 2015/223: {Execution procedure of a spoken command} - G10L 2015/227: {of the speaker; Human-factor methodology} - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING ## Abstract Systems and methods for switching between voice dictation modes using a gesture are provided so that an alternate meaning to a dictated word may be applied. The provided systems and methods time stamp detected gestures and detected words from the voice dictation and compare the time stamp at which a gesture is detected to the time stamp at which a word is detected. When it is determined that a time stamp of a gesture approximately matches a time stamp of a word, the word may be processed to have an alternate meaning, such as a command, punctuation, or action. ## Claims ### Claim 1 A system for processing detected audio waveforms in a first mode and a second mode, comprising: a microphone configured for detecting one or more audio waveforms; a gesture detection sensor configured to detect one or more touchless gestures corresponding to different processing modes; a system clock configured to provide time stamps for the one or more audio waveforms detected by the microphone and time stamps for the one or more touchless gestures detected by the gesture detection sensor, the time stamps for the one or more touchless gestures indicating a starting time and an ending time of each detected touchless gesture; a buffer configured to store the one or more audio waveforms and information regarding the one or more touchless gestures; and one or more processors configured to enter the second mode and selectively process a detected audio waveform stored in the buffer in the second mode when a time stamp of the detected audio waveform is between a first time stamp of a first touchless gesture corresponding to the second mode detected by the gesture detection sensor and a second time stamp associated with one of the one or more touchless gestures. ### Claim 2 (depends on 1) The system of claim 1, wherein the one or more processors are further configured to process a detected audio waveform in the first mode when a time stamp of the detected audio waveform is before the first time stamp or after the second time stamp. ### Claim 3 (depends on 1) The system of claim 1, further comprising a memory storing first meanings for detected audio waveforms processed in the first mode and second meanings for some detected audio waveforms processed in the second mode. ### Claim 4 (depends on 3) The system of claim 3, wherein the second meanings comprise at least one of a command, a punctuation mark, or an action. ### Claim 5 (depends on 4) The system of claim 4, wherein the first meanings comprise text associated with the detected audio waveforms. ### Claim 6 (depends on 1) The system of claim 1, wherein the gesture detection sensor comprises at least one of a camera, an ultrasound transducer, a shadow detector, or an infrared sensor. ### Claim 7 (depends on 1) The system of claim 1, further comprising a display, the display configured to display feedback indicating when the detected audio waveform is being processed in the second mode. ### Claim 8 (depends on 1) The system of claim 1, wherein each of the one or more touchless gestures corresponds to a respective mode of a plurality of modes, and wherein the one or more processors are configured to selectively process an audio waveform detected by the microphone in one mode of the plurality of modes when: a time stamp of the detected audio waveform is between a time stamp corresponding to a start of one touchless gesture of the one or more touchless gestures corresponding to the one mode detected by the gesture detection sensor and a time stamp corresponding to an end of the one touchless gesture; or a time stamp of the detected audio waveform is between the time stamp corresponding to the start of the one touchless gesture and a time stamp corresponding to a start of a next detected touchless gesture. ### Claim 9 (depends on 8) The system of claim 8, wherein the plurality of modes includes at least one of a correction mode, a command mode, or a translation mode. ### Claim 10 (depends on 1) The system of claim 1, wherein the detected audio waveforms are provided to the one or more processors from the buffer for processing based on the one or more touchless gestures detected by the gesture detection sensor. ### Claim 11 (depends on 1) The system of claim 1, wherein the one or more processors are configured to enable a gesture detection procedure using the gesture detection sensor when the microphone detects an audio waveform. ### Claim 12 (depends on 11) The system of claim 11, wherein the gesture detection sensor is inactive until the detected audio waveform is determined to be a human voice. ### Claim 13 (depends on 1) The system of claim 1, wherein the second mode includes one of a correction mode, a command mode, or a translation mode. ### Claim 14 (depends on 1) The system of claim 1, wherein the first time stamp of the first touchless gesture corresponds to a start of the first touchless gesture; and wherein the second time stamp corresponds to an end of the first touchless gesture. ### Claim 15 (depends on 1) The system of claim 1, wherein the first time stamp of the first touchless gesture corresponds to a start of the first touchless gesture; and wherein the second time stamp corresponds to a start of a next detected touchless gesture corresponding to the first mode. ### Claim 16 (depends on 1) The system of claim 1, wherein the time stamps for the one or more audio waveforms are captured at pauses between words. ### Claim 17 (depends on 1) The system of claim 1, wherein the time stamps for the one or more audio waveforms are captured at a beginning and an end of a string including multiple words, and time stamps for individual words are determined by interpolating the time stamps captured at the beginning and the end of the string. ### Claim 18 A method for processing detected audio waveforms in a first mode and a second mode, comprising: detecting, by a microphone, one or more audio waveforms; time stamping the detected one or more audio waveforms; detecting, by a gesture detection sensor, one or more touchless gestures corresponding to one or more modes; time stamping a starting time and an ending time of each of the detected one or more touchless gestures; storing, in a buffer, the one or more audio waveforms and information regarding the one or more touchless gestures; and entering the second mode and selectively processing, by one or more processors coupled to the microphone, an audio waveform of the one or more audio waveforms in the second mode when a time stamp of the audio waveform is between a first time stamp of a first touchless gesture corresponding to the second mode detected by the gesture detection sensor and a second time stamp associated with one of the one or more touchless gestures. ### Claim 19 (depends on 18) The method of claim 18, wherein the first time stamp of the first touchless gesture corresponds to a start of the first touchless gesture; and wherein the second time stamp corresponds to an end of the first touchless gesture or a start of a next detected touchless gesture corresponding to the first mode. ### Claim 20 (depends on 18) The method of claim 18, further comprising processing the audio waveform in the first mode when a time stamp of the audio waveform is before the first time stamp or after the second time stamp. ### Claim 21 (depends on 18) The method of claim 18, wherein processing the audio waveform in the second mode comprises correlating the audio waveform with a meaning that is different from a first meaning corresponding to the first mode. ### Claim 22 (depends on 21) The method of claim 21, wherein the meaning that is different from the first meaning comprises at least one of a command, a punctuation mark, or an action. ### Claim 23 (depends on 22) The method of claim 22, wherein the first meaning comprises text associated with the audio waveform. ### Claim 24 (depends on 18) The method of claim 18, wherein detecting the one or more touchless gestures comprises detecting if the one or more touchless gestures are performed using at least one of a camera, an ultrasound transducer, a shadow detector, or an infrared sensor. ### Claim 25 (depends on 18) The method of claim 18, wherein processing the audio waveform in the second mode comprises generating audio or visual feedback indicating that the audio waveform is being processed in the second mode. ### Claim 26 (depends on 18) The method of claim 18, wherein each of the one or more touchless gestures corresponds to a respective mode of the one or more modes, and wherein selectively processing comprises selectively processing the audio waveform in a mode of the one or more modes corresponding to one touchless gesture of the one or more touchless gestures when: a time stamp of the audio waveform is between a time stamp corresponding to a start of the one touchless gesture detected by the gesture detection sensor and a time stamp corresponding to an end of the one touchless gesture; or a time stamp of the audio waveform is between the time stamp corresponding to the start of the one touchless gesture and a time stamp corresponding to a start of a next detected touchless gesture of the one or more touchless gestures. ### Claim 27 (depends on 18) The method of claim 18, further comprising: providing the one or more audio waveforms from the buffer when the one or more touchless gestures are detected. ### Claim 28 (depends on 18) The method of claim 18, further comprising enabling a gesture detection procedure in response to the microphone detecting an audio waveform. ### Claim 29 A non-transitory computer-readable medium including instructions that, when executed by one or more processors, cause the one or more processors to perform a method for processing detected audio waveforms in a first mode and a second mode, the method comprising: detecting one or more audio waveforms; time stamping the detected one or more audio waveforms; detecting one or more touchless gestures corresponding to one or more modes; time stamping a starting time and an ending time of each of the detected one or more touchless gestures; storing, in a buffer, the one or more audio waveforms and information regarding the one or more gestures; and entering the second mode and selectively processing an audio waveform of the one or more audio waveforms in the second mode when a time stamp of the audio waveform is between a first time stamp of a first touchless gesture of the one or more touchless gestures corresponding to the second mode detected by the gesture detection sensor and a second time stamp associated with one of the one or more touchless gestures. ### Claim 30 A system for processing detected audio waveforms in a first mode and a second mode, comprising: means for detecting one or more audio waveforms and time stamping the one or more audio waveforms; means for detecting one or more touchless gestures corresponding to one or more modes and time stamping a starting time and an ending time of each of the detected one or more touchless gestures; means for storing the one or more audio waveforms and information regarding the one or more gestures; and means for entering the second mode and selectively processing an audio waveform from the one or more audio waveforms in the second mode when a time stamp of the audio waveform is between a first time stamp of a first detected touchless gesture of the one or more touchless gestures corresponding to the second mode and a second time stamp associated with one of the one or more touchless gestures. ### Claim 31 (depends on 30) The system of claim 30, wherein the first time stamp of the first detected touchless gesture corresponds to a start of the first detected touchless gesture; and wherein the second time stamp corresponds to an end of the first detected touchless gesture or a start of a next detected touchless gesture corresponding to the first mode. ### Claim 32 (depends on 30) The system of claim 30, wherein the means for selectively processing processes the audio waveform in the first mode when a time stamp of the audio waveform is before the first time stamp or after the second time stamp. ### Claim 33 (depends on 30) The system of claim 30, wherein the means for selectively processing processes the audio waveform in the second mode by correlating the audio waveform with a second meaning that is different from a first meaning corresponding to the first mode. ### Claim 34 (depends on 33) The system of claim 33, wherein the second meaning that is different from the first meaning comprises at least one of a command, a punctuation mark, or an action. ### Claim 35 (depends on 34) The system of claim 34, wherein the first meaning comprises text associated with the audio waveform input. ### Claim 36 (depends on 30) The system of claim 30, wherein the means for detecting the one or more touchless gestures comprises at least one of a camera, an ultrasound transducer, a shadow detector, or an infrared sensor. ### Claim 37 (depends on 30) The system of claim 30, wherein the means for selectively processing generates audio or visual feedback indicating that the audio waveform is being processed in the second mode. ### Claim 38 (depends on 30) The system of claim 30, wherein each of the one or more touchless gestures corresponding to a respective mode of the one or more modes, and wherein the means for selectively processing comprises selectively processing the audio waveform in a mode of the one or more modes corresponding to one touchless gesture of the one or more touchless gestures when: a time stamp of the audio waveform is between a time stamp corresponding to a start of the one touchless gesture detected by the gesture detection sensor and a time stamp corresponding to an end of the one touchless gesture; or a time stamp of the audio waveform is between the time stamp corresponding to the start of the one touchless gesture and a time stamp corresponding to a start of a next detected touchless gesture of the one or more touchless gestures. ### Claim 39 (depends on 30) The system of claim 30, wherein the means for storing the audio waveforms provides the stored audio waveforms to the means for selectively processing when the means for detecting the one or more touchless gestures detects the one or more touchless gestures.