Multimodal interface for input of text
OwnerAshwin P. Rao, Gregory M. Aronov +1inventors
InventorsAshwin P. Rao, Gregory M. Aronov, Marat V. Garafutdinov
Life of this patent
Estimating the term…
Abstract
The disclosure describes an overall system/method for text-input using a multimodal interface with a combination of speech recognition and text prediction. Specifically, an “always listening” mode for entering words is combined with a push-to-speak mode for entering symbols and phrases. In addition, these two modes are further combined with keypad based text prediction. Finally, the overall user-interface of the proposed system is designed such that it enhances existing standard text-input methods; thereby minimizing the behavior change for mobile users.
Claims17
A multimodal system, the system comprising:
- a speech detection module configured to detect audio;
- a speech recognition module configured to accept the audio and at least one letter input, to dynamically create a reduced dictionary based on the at least one letter input, to search the reduced dictionary for a set of speech recognized words based on the detected audio, and to determine an acoustic score for each of the speech recognized words;
- a text prediction module configured to accept the least one letter input, to predict a set of predicted words associated with the at least one letter input and to determine a language model score for each of the predicted words;
- and
- a combine module configured to accept the set of predicted words and the set of speech recognized words and to create a set of choice words, wherein the set of choice words is based upon a dynamic weight determined for each of the speech recognized words and each of the predicted words using the respective acoustic score and language model score, wherein the set of choice words is output to aid in a determination of a subsequent user-action.
A computer-implemented method for multimodal text recognition, comprising:
- a) detecting an utterance that is spoken;
- b) extracting at least one prosodic feature associated with the utterance;
- c) dynamically reducing a dictionary based on the at least one prosodic feature and searching the dictionary for at least one entry starting with the first letter typed;
- d) determining a set of potential matches from the at least one entry and determining an acoustic score related to each potential match in the set of potential matches;
- e) determining a set of predicted matches using a text prediction technique based on the first letter typed and determining a language model score related to each of the predicted matches within the set of predicted matches;
- f) combining the set of potential matches with the set of predicted matches into a combined list by dynamically weighing the potential matches and the predicted matches based on their corresponding acoustic scores and language model scores, respectively;
- g) determining a set of choice words based on a confidence value determined for each entry in the combined list.
A computing device configured to provide a multimodal interface for entering text, the computing device comprising:
- a computer storage medium including computer-readable instructions:
- a processor configured by the computer-readable instructions to:
- provide a user-interface for entering and displaying text;
- detect audio when the user-interface is in a listening mode;
- recognize input of a first letter into a message window of the user-interface;
- dynamically reduce a search dictionary based on the first letter;
- search the search dictionary based on the detected audio to determine a set of speech recognized words;
- predict a set of predicted words based on the first letter and determine a language model score for each of the predicted words;
- create a set of choice words based upon a dynamic weight determined for each of the speech recognized words and each of the predicted words using the respective acoustic score and language model score of the predicted words and the speech recognized words, respectively;
- and
- display the set of choice words in a choice window of the user-interface to aid in a determination of a subsequent user-action, wherein the subsequent user-action provides usage data that affects the dynamic reduction of the search dictionary.
Dependencies are inferred from the claim text.