System and method for continuous multimodal speech and gesture interaction

A system comprising:

• a processor;
• and
• a computer-readable storage medium storing instructions which, when executed by the processor, cause the processor to perform operations comprising:
• monitoring an audio stream associated with a non-tactile gesture input stream;
• identifying a speech event in an audio stream;
• analyzing data from the non-tactile gesture input stream to identify a non-tactile gesture event;
• determining, based on the non-tactile gesture event, a temporal window associated with a time of the speech event;
• and
• processing the speech event and the non-tactile gesture event within the temporal window to produce a multimodal command, wherein the temporal window comprises one of a modified temporal window representing a change from an original temporal window associated with the speech event or the original temporal window associated with the speech event and which is unchanged based on a type of the non-tactile gesture event.

A computer-readable storage device storing instructions which, when executed by a processor, cause the processor to perform operations comprising:

• monitoring an audio stream associated with a non-tactile gesture input stream;
• identifying a speech event in an audio stream;
• analyzing data from the non-tactile gesture input stream to identify a non-tactile gesture event;
• determining, based on the non-tactile gesture event, a temporal window associated with a time of the speech event;
• and
• processing the speech event and the non-tactile gesture event within the temporal window to produce a multimodal command, wherein the temporal window comprises one of a modified temporal window representing a change from an original temporal window associated with the speech event or the original temporal window associated with the speech event and which is unchanged based on a type of the non-tactile gesture event.