Multi-modal fusion engine

An electronic device, comprising:

• a first input device that detects user input using a first modality;
• a second input device that detects user input using a different modality;
• a processor;
• and
• a memory storing instructions executable by the processor to:
• detect, using a first uni-modal input processor, and at a first time, a command input using a first modality and associating a command input time stamp with the command input, wherein the command input comprises a command and at least one deictic expression;
• detect, within an application, using a second uni-modal input processor, and at a second time different than the first time, a selecting input associated with an entity within the application and using a different modality, wherein the detecting comprises identifying a location on the electronic device corresponding to a location of the selecting input;
• store the selecting input as contextual data with an associated selecting input time stamp;
• combine, by receiving the command input from the first uni-modal input processor and the selecting input from the second uni-modal input processor, the command input and the selecting input into an application command having at least one application parameter referencing the entity of the selecting input;
• the combining comprising associating the command input with the selecting input based upon the contextual data and resolving the at least one deictic expression into the at least one application parameter by matching the at least one deictic expression to one or more entities contained within the application using a resolver module that utilizes the contextual data and the identified location of the selecting input, associated selecting input time stamp, and the command input time stamp to identify the entity of the selecting input;
• and
• provide the application command to an application running on the electronic device for execution.

A program product, comprising:

• a storage device storing processor executable code, comprising:
• code that detects, using a first uni-modal input processor, and at a first time, a command input using a first modality and associating a command input time stamp with the command input, wherein the command input comprises a command and at least one deictic expression;
• code that detects, within an application, using a second uni-modal input processor, and at a second time different than the first time, a selecting input associated with an entity within the application and using a different modality, wherein the detecting comprises identifying a location on the electronic device corresponding to a location of the selecting input;
• code that stores the selecting input as contextual data with an associated selecting input time stamp;
• code that combines, by receiving the command input from the first uni-modal input processor and the selecting input from the second uni-modal input processor, the command input and the selecting input into an application command having at least one application parameter referencing the entity of the selecting input;
• the combining comprising associating the command input with the selecting input based upon the contextual data and resolving the at least one deictic expression into the at least one application parameter by matching the at least one deictic expression to one or more entities contained within the application using a resolver module that utilizes the contextual data and the identified location of the selecting input, associated selecting input time stamp, and the command input time stamp to identify the entity of the selecting input;
• and
• code that provides the application command to an application running on the electronic device for execution.