Systems and methods for improving autocorrect accuracy by fusing keystroke input with real-time lip-movement cues and optional whisper-level audio

A computer-implemented method for generating correction candidates for text input, the method comprising:

• receiving, by one or more processors, a typed string entered through a keyboard-based text-entry interface;
• receiving lip-movement evidence captured by an imaging sensor while or in association with entry of the typed string;
• generating, from the typed string alone, a standard autocorrect candidate list comprising one or more candidate text strings;
• deriving, from the lip-movement evidence, one or more visual speech cues comprising a syllable count, a coarse phoneme sequence, a viseme sequence, a partial viseme sequence, a mouth-shape cue, or another lip-derived candidate cue;
• generating, based at least in part on the typed string, a text-derived candidate pool;
• using the one or more visual speech cues before or during candidate generation to retrieve, add, or inject one or more candidate text strings, including at least one candidate text string that was not present in the standard autocorrect candidate list generated from the typed string alone;
• forming a fused candidate set comprising candidates derived from the typed string and candidates retrieved, added, or injected using the one or more visual speech cues;
• scoring candidate text strings in the fused candidate set using both text-derived evidence and lip-derived evidence;
• and
• outputting, through the keyboard-based text-entry interface, a correction result for the typed string based at least in part on the scoring.

A text-entry correction system comprising:

• one or more processors;
• and
• one or more memories storing instructions that, when executed by the one or more processors, cause the text-entry correction system to:
• receive typed input through a keyboard-based text-entry interface;
• receive lip-movement evidence from an imaging sensor when a lip-movement input channel is active;
• receive whisper-level audio evidence from a microphone or audio interface when a whisper-level audio input channel is active;
• operate a shared autocorrect or spell-correction pipeline using one, two, or all of a typed-input channel, the lip-movement input channel, and the whisper-level audio input channel without requiring explicit user toggling between a typing mode, a mouthing mode, a whispering mode, or a mixed-input mode;
• when the typed-input channel is active, generate or receive candidate text strings based at least in part on the typed input;
• when the lip-movement input channel is active, generate lip-derived evidence for candidate scoring, candidate reordering, candidate retrieval, candidate pruning, or candidate injection;
• when the whisper-level audio input channel is active, generate audio-derived evidence for candidate scoring, candidate reordering, candidate retrieval, candidate pruning, or candidate injection;
• combine available evidence from the active input channels to produce a correction result for a current typed string or current text-entry state;
• and
• output the correction result through the keyboard-based text-entry interface as a suggestion, reordered candidate list, or automatic replacement.

A computer-implemented method for correcting text input using an existing spell-correction engine, the method comprising:

• receiving, by one or more processors, a current typed string entered through a keyboard-based text-entry interface;
• receiving lip-movement evidence, whisper-level audio evidence, or both, while or in association with entry of the current typed string;
• deriving, from the lip-movement evidence, the whisper-level audio evidence, or both, a phoneme, viseme, syllable, pronunciation-derived, or visual speech representation associated with an intended text string;
• generating, from the phoneme, viseme, syllable, pronunciation-derived, or visual speech representation, a bounded set of plausible misspellings or orthographic proxy strings;
• submitting the current typed string and the bounded set of plausible misspellings or orthographic proxy strings to the existing spell-correction engine;
• receiving, from the existing spell-correction engine, one or more recovered candidate text strings;
• and
• outputting, through the keyboard-based text-entry interface, a correction result based at least in part on the one or more recovered candidate text strings.