# US 8,571,862 B2 · Multimodal interface for input of text

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 8,571,862 B2
- Document ID: US8571862B2
- Status: Granted
- Kind: B2
- Application number: 12/578,355
- Filed: 2009-10-13
- Published: 2010-02-04
- Granted: 2013-10-29
- Earliest claimed: 2006-11-30
- Applicants: Ashwin P. Rao; Gregory M. Aronov; Marat V. Garafutdinov
- Inventors: Ashwin P. Rao; Gregory M. Aronov; Marat V. Garafutdinov
- Web page: https://patentgrep.com/patent/US8571862B2

## CPC

- G10L 15/24: Speech recognition using non-acoustical features
- G06F 3/0237: {using prediction or retrieval techniques}
- G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue
- G10L 15/32: Multiple recognisers used in sequence or in parallel; Score combination systems therefor, e.g. voting systems
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

The disclosure describes an overall system/method for text-input using a multimodal interface with a combination of speech recognition and text prediction. Specifically, an “always listening” mode for entering words is combined with a push-to-speak mode for entering symbols and phrases. In addition, these two modes are further combined with keypad based text prediction. Finally, the overall user-interface of the proposed system is designed such that it enhances existing standard text-input methods; thereby minimizing the behavior change for mobile users.

## Claims

### Claim 1

A multimodal system, the system comprising:
a speech detection module configured to detect audio;
a speech recognition module configured to accept the audio and at least one letter input, to dynamically create a reduced dictionary based on the at least one letter input, to search the reduced dictionary for a set of speech recognized words based on the detected audio, and to determine an acoustic score for each of the speech recognized words;
a text prediction module configured to accept the least one letter input, to predict a set of predicted words associated with the at least one letter input and to determine a language model score for each of the predicted words; and
a combine module configured to accept the set of predicted words and the set of speech recognized words and to create a set of choice words, wherein the set of choice words is based upon a dynamic weight determined for each of the speech recognized words and each of the predicted words using the respective acoustic score and language model score, wherein the set of choice words is output to aid in a determination of a subsequent user-action.

### Claim 2 (depends on 1)

The multimodal system of claim 1, wherein the subsequent user-action provides usage data that the speech recognition module takes into account when dynamically creating the reduced dictionary.

### Claim 3 (depends on 1)

The multimodal system of claim 1, further comprising a user-interface module configured to accept an interaction by a user with respect to the choice words.

### Claim 4 (depends on 3)

The multimodal system of claim 3, wherein the interaction by the user initiates another cycle of text prediction, speech recognition, and combining to obtain another set of choice words.

### Claim 5 (depends on 1)

The multimodal system of claim 1, further comprising an adaption module configured to determine an adaption score based on a usage pattern, and the set of choice words being further based on the adaption score.

### Claim 6 (depends on 5)

The multimodal system of claim 5, wherein the usage pattern is based on a selected word from the set of choice words.

### Claim 7

A computer-implemented method for multimodal text recognition, comprising:
a) detecting an utterance that is spoken;
b) extracting at least one prosodic feature associated with the utterance;
c) dynamically reducing a dictionary based on the at least one prosodic feature and searching the dictionary for at least one entry starting with the first letter typed;
d) determining a set of potential matches from the at least one entry and determining an acoustic score related to each potential match in the set of potential matches;
e) determining a set of predicted matches using a text prediction technique based on the first letter typed and determining a language model score related to each of the predicted matches within the set of predicted matches;
f) combining the set of potential matches with the set of predicted matches into a combined list by dynamically weighing the potential matches and the predicted matches based on their corresponding acoustic scores and language model scores, respectively;
g) determining a set of choice words based on a confidence value determined for each entry in the combined list.

### Claim 8 (depends on 7)

The computer-implemented method of claim 7, further comprising dynamically reducing the dictionary based on the first letter typed.

### Claim 9 (depends on 7)

The computer-implemented method of claim 7, further comprising providing the set of choice words to an application.

### Claim 10 (depends on 7)

The computer-implemented method of claim 7, further comprising accepting an interaction by a user with respect to the set of choice words.

### Claim 11 (depends on 10)

The computer-implemented method of claim 10, wherein the interaction by the user initiates processing to perform steps c-g to obtain another set of choice words.

### Claim 12

A computing device configured to provide a multimodal interface for entering text, the computing device comprising:
a computer storage medium including computer-readable instructions:
a processor configured by the computer-readable instructions to:
provide a user-interface for entering and displaying text;
detect audio when the user-interface is in a listening mode;
recognize input of a first letter into a message window of the user-interface;
dynamically reduce a search dictionary based on the first letter;
search the search dictionary based on the detected audio to determine a set of speech recognized words;
predict a set of predicted words based on the first letter and determine a language model score for each of the predicted words;
create a set of choice words based upon a dynamic weight determined for each of the speech recognized words and each of the predicted words using the respective acoustic score and language model score of the predicted words and the speech recognized words, respectively; and
display the set of choice words in a choice window of the user-interface to aid in a determination of a subsequent user-action, wherein the subsequent user-action provides usage data that affects the dynamic reduction of the search dictionary.

### Claim 13 (depends on 12)

The computing device of claim 12, further comprising a soft-key on the user-interface for switching modes.

### Claim 14 (depends on 13)

The computing device of claim 13, wherein the mode comprises a symbol mode for inputting a spoken word for a symbol and selecting one of a set of characters displayed on the user-interface for the symbol.

### Claim 15 (depends on 13)

The computing device of claim 13, wherein the mode comprises a phrase mode for inputting spoken words for a phrase and selecting one of a set of phrases displayed on the user-interface based on the first letter.

### Claim 16 (depends on 15)

The computing device of claim 15, wherein the processor is further configured by the computer-readable instructions to recognize input of a second letter and modify the set of phrases displayed on the user-interface based on the first and second letter.

### Claim 17 (depends on 12)

The computing device of claim 12, wherein the user-interface further comprises a menu item to switch between a “Push-to-Speak” mode and a “Listening” mode for detecting the audio.