US 6,285,785 B1 · Message recognition employing integrated speech and handwriting information

Abstract

A method of, and apparatus for, operating an automatic message recognition system. In accordance with the method the following steps are executed: a user's speech is converted to a first signal; a user's handwriting is converted to a second signal; and the first signal and the second signal are processed to decode a consistent message, conveyed separately by the first signal and by the second signal, or conveyed jointly by the first signal and the second signal. The step of processing includes the steps of converting the first signal into a plurality of first multi-dimensional vectors and converting the second signal into a plurality of second multi-dimensional vectors. For a system employing a combined use of speech and handwriting the step of processing includes a further step of combining individual ones of the plurality of first multi-dimensional vectors and individual ones of the plurality of second multi-dimensional vectors to form a plurality of third multi-dimensional vectors. The multi-dimensional vectors are employed to train a single set of word models, for joint use of speech and handwriting, or two sets of word models, for sequentially employed or merged speech and handwriting.

Claims46

Dependent claims

A message recognition system comprising:

a first transducer for converting a user's speech to a first signal;
a second transducer for converting the user's handwriting to a second signal;
and
a data processor, having a first input coupled to the first signal and a second input coupled to the second signal, for processing the first signal and the second signal to identify an informational content of the first signal and the second signal, said data processor including,
a first likelihood estimator for generating a first list comprised of one or more probable messages conveyed by the informational content of the first signal;
a second likelihood estimator for generating a second list comprised of one or more probable messages conveyed by the information content of the second signal;
wherein a probable message is comprised of at least one word;
a likelihood merger for selectively merging the first list and the second list to form a third list;
a decoder for selecting from the third list a most probable one of the probable messages to be an output message;
and
means for outputting the output message.

8 more elements

A method for operating a message recognition system, comprising the steps of:

operating a first transducer for converting a user's speech to a first signal;
operating a second transducer for converting the user's handwriting to a second signal;
and
operating a digital data processor for processing the first signal and the second signal to identify an informational content of a consistent message that is conveyed separately by the first signal and by the second signal or that is conveyed jointly by the first signal and by the second signal, the step of processing including the steps of,
operating at least one likelihood estimator for generating one or more probable messages conveyed by the informational content of both the first signal and the second signal, wherein a probable message is comprised of at least one word;
and
operating a decoder for selecting a most probable one of the probable messages to be an output message, wherein the step of operating a decoder includes a step of considering a first weighting value, that is associated with the first signal, and a second weighting value, that is associated with the second signal.

4 more elements

A method of operating a message recognition system, comprising the steps of:

operating a first transducer for converting a user's speech to a first signal;
operating a second transducer for converting, in parallel with the step of converting a user's speech, the user's handwriting to a second signal;
combining with a digital data processor the first signal and the second signal to generate a third signal;
operating a likelihood estimator for generating a list comprised of one or more probable messages conveyed by the informational content of the third signal, wherein a probable message is comprised of at least one word;
and
operating a decoder for selecting from the list a most probable one of the probable messages to be an output message.

3 more elements

A message recognition system that includes a digital data processor, said digital data processor comprising:

a user interface having a first input coupled to an output of a speech transducer means and a second input coupled to an output of a handwriting transducer means, for receiving signals therefrom and for converting the signals to a first multi-dimensional representation of a speech signal and to a second multi-dimensional representation of a handwriting signal;
a first likelihood estimator, having an input coupled to said first multi-dimensional representation of the speech signal, for generating, in accordance with an associated first word model and in response to the first multi-dimensional representation, a first list comprised of one or more probable words that the first multi-dimensional representation may represent;
a second likelihood estimator, having an input coupled to said second multi-dimensional representation of the handwriting signal, for generating, in accordance with an associated second word model and in response to the second multi-dimensional representation, a second list comprised of one or more probable words that the second multi-dimensional representation may represent;
a likelihood merger, having an input coupled to an output of said first generating means and to an output of said second generating means, for selectively merging said first list and said second list into a third list comprised of probable words;
and
a decoder that is responsive to the probable words of said third list, for selecting a most probable word as an output word.