# US 8,751,217 B2 · Multi-modal input on an electronic device > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 8,751,217 B2 - Document ID: US8751217B2 - Status: Granted - Kind: B2 - Application number: 13/249,172 - Filed: 2011-09-29 - Published: 2012-01-26 - Granted: 2014-06-10 - Earliest claimed: 2009-12-23 - Assignees: Google Inc. - Applicants: Brandon M. Ballinger; Johan Schalkwyk; Michael H. Cohen; William J. Byrne; Gudmundur Hafsteinsson; Michael J. LeBeau - Inventors: Brandon M. Ballinger; Johan Schalkwyk; Michael H. Cohen; William J. Byrne; Gudmundur Hafsteinsson; Michael J. LeBeau - Web page: https://patentgrep.com/patent/US8751217B2 ## CPC - G06F 3/167: {Audio in a user interface, e.g. using voice commands for navigating, audio feedback} - G06F 3/04886: by partitioning the display area of the touch-screen or the surface of the digitising tablet into independently controllable areas, e.g. virtual keyboards or menus - G06F 40/284: Lexical analysis, e.g. tokenisation or collocates - G06F 40/58: Use of machine translation, e.g. for multi-lingual retrieval, for server-side translation for client devices or for real-time translation - G10L 15/005: {Language recognition} - G10L 15/18: using natural language modelling - G10L 15/183: using context dependencies, e.g. language models - G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue - G10L 15/26: Speech to text systems (G10L15/08 takes precedence) - G10L 15/30: Distributed recognition, e.g. in client-server systems, for mobile phones or network applications - G10L 15/197: Probabilistic grammars, e.g. word n-grams - G10L 2015/223: {Execution procedure of a spoken command} - G10L 2015/228: {of application context} - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING ## Abstract A computer-implemented input-method editor process includes receiving a request from a user for an application-independent input method editor having written and spoken input capabilities, identifying that the user is about to provide spoken input to the application-independent input method editor, and receiving a spoken input from the user. The spoken input corresponds to input to an application and is converted to text that represents the spoken input. The text is provided as input to the application. ## Claims ### Claim 1 A computer-implemented input-method editor process comprising: receiving a request from a user of an electronic device for an application-independent input method editor having written and spoken input capabilities, wherein the application-independent input method editor is configured to receive input for a plurality of applications executable by the electronic device; identifying that the user is about to provide spoken input to the application-independent input method editor; receiving a spoken input from the user, wherein the spoken input corresponds to an input to an application from the plurality of applications; generating, with the electronic device, a correlation between the spoken input and the application, from the plurality of applications, to which the spoken input is determined to correspond, wherein generating the correlation comprises (i) determining an application type for the application that identifies a category of application to which the application is assigned, each category of application capable of including a plurality of different applications, and (ii) matching one or more utterances in the spoken input to one or more stored terms that are associated with the application type; providing the spoken input and the generated correlation to a server, wherein the server is remote to the electronic device and includes a speech recognition system configured to access one or more language models for recognizing text based on the spoken input and the correlation; receiving text from the server, wherein the text represents a translation of the spoken input; selecting an application, from the plurality of applications, to receive the text, the selection using the generated correlation; and providing the text as an input to the selected application. ### Claim 2 (depends on 1) The process of claim 1, further comprising: presenting a list of candidates of text representing the spoken input to the user; and receiving a selection from the user of a candidate from the list; wherein providing the text to the application as user input comprises providing the selection of the candidate from the list to the application. ### Claim 3 (depends on 1) The process of claim 1, further comprising: receiving a written input from the user; determining a language of the written input; and providing a language indicator to the server based on the determined language indicating the language of the spoken input. ### Claim 4 (depends on 1) The process of claim 1, further comprising: providing a context indicator to the server such that the speech recognition system can select a language model from a plurality of language models based on the context indicator, where the context indicator specifies the context in which the user input is received. ### Claim 5 (depends on 4) The process of claim 4, wherein the context indicator specifies a webpage in which the user input is received. ### Claim 6 (depends on 4) The process of claim 4, wherein the context indicator specifies an application in which the user input is received. ### Claim 7 (depends on 4) The process of claim 4, wherein the context indicator specifies a web form in which the user input is received. ### Claim 8 (depends on 7) The process of claim 7, wherein the context indicator further specifies a field in the web form in which the user input is received. ### Claim 9 (depends on 8) The process of claim 8, wherein the context indicator further specifies metadata associated with the field in the web form. ### Claim 10 (depends on 1) The process of claim 1, further comprising: receiving a written input from the user in a first writing system; and presenting one or more candidates in a second writing system based on the written input in the first writing system. ### Claim 11 (depends on 10) The process of claim 10, wherein the first writing system is Pinyin and the second writing system is Hanzi. ### Claim 12 (depends on 1) The process of claim 1, further comprising: performing intermediate processing on the received spoken input; wherein providing the spoken input to a server comprises providing the intermediately processed spoken input to the server. ### Claim 13 A system comprising: a data processing apparatus; and a data store coupled to the data processing apparatus, in which is stored: an application-independent input method editor configured to receive input for a plurality of applications executable by an electronic device, the application-independent input method editor operable to: receive spoken input from a user of the electronic device that is directed to an application of the plurality of applications; generating, with the electronic device, a correlation between the spoken input and the application, from the plurality of applications, to which the spoken input is determined to correspond, wherein generating the correlation comprises (i) determining an application type for the application that identifies a category of application to which the application is assigned, each category of application capable of including a plurality of different applications, and (ii) matching one or more utterances in the spoken input to one or more stored terms that are associated with the application type; provide the spoken input and the generated correlation to a server, wherein the server is remote to the electronic device and includes a speech recognition system configured to access one or more language models for recognizing text based on the spoken input and the correlation; receive text from the server, wherein the text represents a translation of the spoken input; select an application, from the plurality of applications, to receive the text, the selection using the generated correlation; and provide the text as input to the selected application. ### Claim 14 (depends on 13) The system of claim 13, wherein the application-independent input method editor is further operable to: receive a written input from the user; determine a language of the written input; and provide a language indicator to the server based on the determined language indicating the language of the spoken input. ### Claim 15 (depends on 13) The system of claim 13, wherein the application-independent input method editor is further operable to: provide a context indicator to the server such that the speech recognition system can select a language model from a plurality of language models based on the context indicator, where the context indicator specifies the context in which the user input is received. ### Claim 16 (depends on 15) The system of claim 15, wherein the context indicator specifies an application from the plurality of applications in which the user input is received. ### Claim 17 (depends on 13) The system of claim 13, wherein the application-independent input method editor is further operable to: perform intermediate processing on the received spoken input; wherein providing the spoken input to a server comprises providing the intermediately processed spoken input to the server. ### Claim 18 A non-transitory computer-readable storage device encoded with a computer program product, the computer program product including instructions that, when executed, cause data processing apparatus to perform operations comprising: receiving spoken input from a user of an electronic device in an application of the plurality of applications; generating, with the electronic device, a correlation between the spoken input and the application, from the plurality of applications, to which the spoken input is determined to correspond, wherein generating the correlation comprises (i) determining an application type for the application that identifies a category of application to which the application is assigned, each category of application capable of including a plurality of different applications, and (ii) matching one or more utterances in the spoken input to one or more stored terms that are associated with the application type; providing the spoken input and the generated correlation to a server, wherein the server is remote to the electronic device and includes a speech recognition system configured to access one or more language models for recognizing text based on the spoken input and the correlation; receiving text from the server, wherein the text represents a translation of the spoken input; selecting an application, from the plurality of applications, to receive the text, the selection using the generated correlation; and providing the text as input to the application. ### Claim 19 (depends on 18) The non-transitory computer-readable storage device of claim 18, wherein the instructions are further operable to: receive a written input from the user; determine a language of the written input; and provide a language indicator to the server based on the determined language indicating the language of the spoken input. ### Claim 20 (depends on 18) The non-transitory computer-readable storage device of claim 18, wherein the instructions are further operable to: provide a context indicator to the server such that the speech recognition system can select a language model from a plurality of language models based on the context indicator, where the context indicator specifies the context in which the user input is received. ### Claim 21 (depends on 20) The non-transitory computer-readable storage device of claim 20, wherein the context indicator specifies an application from the plurality of applications in which the user input is received. ### Claim 22 (depends on 18) The non-transitory computer-readable storage device of claim 18, wherein the instructions are further operable to: perform intermediate processing on the received spoken input; wherein providing the spoken input to a server comprises providing the intermediately processed spoken input to the server.