# US 9,519,353 B2 · Combined speech and touch input for observation symbol mappings > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 9,519,353 B2 - Document ID: US9519353B2 - Status: Granted - Kind: B2 - Application number: 12/413,683 - Filed: 2009-03-30 - Published: 2010-09-30 - Granted: 2016-12-13 - Earliest claimed: 2009-03-30 - Assignees: Symbol Technologies, Inc. - Applicants: Bruce Willins - Inventors: Bruce Willins - Web page: https://patentgrep.com/patent/US9519353B2 ## CPC - G06F 3/0237: {using prediction or retrieval techniques} - G06F 3/04886: by partitioning the display area of the touch-screen or the surface of the digitising tablet into independently controllable areas, e.g. virtual keyboards or menus - G06F 40/274: Converting codes to words; Guess-ahead of partial word inputs - G10L 15/20: Speech recognition techniques specially adapted for robustness in adverse environments, e.g. in noise, of stress induced speech (G10L21/02 takes precedence) - G10L 15/24: Speech recognition using non-acoustical features - G10L 15/142: {Hidden Markov Models [HMMs]} - G10L 15/26: Speech to text systems (G10L15/08 takes precedence) - G10L 2015/226: {using non-speech characteristics} - H04M 1/271: {controlled by voice recognition} - H04M 2250/74: with voice recognition means - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING - H04M: TELEPHONIC COMMUNICATION (circuits for controlling other apparatus via a telephone cable and not involving telephone switching apparatus G08) ## Abstract The invention relates to systems and or methodologies for enabling combined speech and touch inputs for observation symbol mappings. More particularly, the current innovation leverages the commonality of touch screen display text entry and speech recognition based text entry to increase the speed and accuracy of text entry via mobile devices. Touch screen devices often contain small and closely grouped keypads that can make it difficult for a user to press the intended character, by combining touch screen based text entry with speech recognition based text entry the aforementioned limitation can be overcome efficiently and conveniently. ## Claims ### Claim 1 A system for mobile device text entry, comprising: memory storing computer readable instructions configured for executing a plurality of components; and a processor, coupled to the memory and operable to execute the instructions, the plurality of components comprising: a text recognition component that obtains a set of inputs from a text entry component, determines a set of digitized coordinates for the set of inputs, and produces a first set of observations based on a set of possible intended user inputs and the digitized coordinates, wherein the text entry component is a touch screen display; a voice recognition component that obtains audio data from an audio capture component, and produces a second set of observations based on the audio data, wherein the audio capture component obtains the audio data concurrently as the text entry component obtains the set of inputs; and a predictive component that combines the first set of observations and the second set of observations into a concatenated observation sequence, and generates a textual output based on the concatenated observation sequence, wherein the generated textual output includes a first predetermined character that is predicted by the predictive component based on (a) a substitute character received by the text entry component, the substitute character is different from the first predetermined character, and (b) pronunciation of the first predetermined character received by the audio capture component. ### Claim 2 (depends on 1) The system of claim 1, wherein the touch screen display provides a virtual set of keys for a user to input a set of characters. ### Claim 3 (depends on 1) The system of claim 1, wherein the user triggers an input via the text entry component that enables a secondary input mode, wherein the secondary input mode includes speech recognition. ### Claim 4 (depends on 1) The system of claim 1 wherein the predictive component generates the textual output based on the sets in the concatenated observation sequence by employing at least one of a Hidden Markov model and a Viterbi decoder that processes the concatenated observation sequence. ### Claim 5 (depends on 1) The system of claim 1 wherein the first predetermined character is a special character. ### Claim 6 A method for mobile device text entry, comprising: generating a first set of observations based on a set of possible intended user inputs and a determined set of digitized coordinates associated with the set of possible intended user inputs obtained via a text input device, wherein the text input device is a touch screen display; generating a second set of observations based on user inputs obtained via an audio input device, wherein user inputs via the text input device and the audio input device are obtained concurrently; combining the first set of observations and the second set of observations to produce a set of combined observations; and producing a textual output by processing the set of combined observations, wherein the produced textual output includes a first predetermined character that is predicted based on (a) a substitute character received by the text input device, the substitute character is different from the first predetermined character, and (b) pronunciation of the first predetermined character received by the audio input device. ### Claim 7 (depends on 6) The method of claim 6, wherein the touch screen display provides a graphical key wedge which enables users to select at least one character by touching the character on the display. ### Claim 8 (depends on 6) The method of claim 6, further comprising triggering an input that enables an alternative input mode. ### Claim 9 (depends on 8) The method of claim 8, wherein the alternative input mode includes speech recognition. ### Claim 10 (depends on 8) The method of claim 8, further comprising enabling the user to access inputs via the alternative input mode that are not currently available. ### Claim 11 (depends on 6) The method of claim 6 wherein the textual output is produced by processing the set of combined observations employing at least one of a Hidden Markov model and a Viterbi decoder. ### Claim 12 (depends on 6) The method of claim 6 wherein the first predetermined character is a special character. ### Claim 13 A system for mobile device text entry, comprising: means for generating a first set of observations based on a set of possible intended user inputs and a determined set of digitized coordinates associated with the set of possible intended user inputs obtained via a touch screen display that provides a graphical key interface, wherein users select at least one unintended character by pressing at a location on the graphical key interface proximate to the location of the unintended character; means for generating a second set of observations based on user inputs obtained via an audio capture device, wherein user inputs via the touch screen display and the audio capture device are obtained concurrently; and means for combining the first set of observations and the second set of observations to produce a set of concatenated observations; and means for producing a textual output based on the set of concatenated observations, wherein the produced textual output includes a first predetermined character that is predicted based on (a) a substitute character received by the graphical key interface, the substitute character is different from the first predetermined character, and (b) pronunciation of the first predetermined character received by the audio capture device. ### Claim 14 (depends on 13) The system of claim 13, further comprising means for triggering an input that enables an alternative input mode, wherein the alternative input mode provides the user with input options that are not presently available. ### Claim 15 (depends on 13) The system of claim 13 wherein the textual output is produced by using at least one of a Viterbi decoder and a Hidden-Markov Model. ### Claim 16 (depends on 13) The system of claim 13 wherein the first predetermined character is a special character.