Method and system for resolving cross-modal references in user inputs

A method for resolving cross-modal references in user inputs to a data processing system, the user inputs being entered through at least one input modality, the data processing system generating references based on each user input, each reference comprising at least one reference variable, the method comprising:

• collecting multimodal interpretations (MMIs) corresponding to the user inputs for a user turn;
• classifying the collected MMIs into one or more sets of semantically compatible MMIs;
• identifying MMIs that comprise one or more references in each of the one or more sets of semantically compatible MMIs;
• creating one or more reference association structures (RASs) for each set of semantically compatible MMIs, one RAS for each unique type of MMI required to resolve the references in the identified MMIs with the set of semantically compatible MMIs;
• mapping the reference variables of the references in the identified MMIs of a set of semantically compatible MMIs to the one or more RASs contained in that set of semantically compatible MMIs, the mapping being based on the type of MMI required by the reference variable;
• sorting the reference variables within each RAS using one or more pre-determined criteria;
• mapping each referent, which is an MMI that does not have reference variables, of a set of semantically compatible MMIs to an RAS contained in the set of semantically compatible MMIs requiring referents that are of the same type or super type as the referent;
• sorting the referents in each RAS using the one or more pre-determined criteria;
• and
• binding the reference variables in each RAS to one or more referents in the RAS.

A method for resolving cross-modal references in user inputs to a data processing system, the user inputs being entered through at least one input modality, the data processing system generating references based on each user input, each reference comprising at least one reference variable, the method comprising:

• segmenting the user inputs, wherein the segmenting comprises collecting a set of multimodal interpretations (MMIs) corresponding to the user inputs for a user turn;
• classifying the collected set of MMIs semantically, wherein semantically classifying the collected set of MMIs comprises creating sets of joint MMIs, each set of joint MMIs comprising MMIs of semantically compatible types;
• resolving the reference variables in the sets of joint MMIs to create corresponding sets of reference-resolved MMIs, wherein resolving the reference variables comprises replacing each reference variable with a resolved value;
• and
• integrating the set of reference-resolved MMIs to generate a corresponding set of integrated MMIs.

An electronic equipment that resolves cross-modal references in user inputs to a data processing system, the user inputs being entered through at least one input modality, the equipment comprising:

• means for generating a set of multimodal interpretations (MMIs) based on the user inputs collected during a turn, at least one MMI comprising at least one reference, each reference comprising at least one reference variable;
• means for generating one or more sets of joint MMIs, each set of joint MMIs comprising MMIs of semantically compatible types;
• means for generating a set of reference resolved MMIs for each set of joint MMIs, wherein the generation of the set of reference resolved MMIs is done by resolving reference variables of the references of the set of joint MMIs;
• and
• means for generating an integrated MMI for each set of reference resolved MMIs, wherein the generation of the integrated MMI is done by unifying the set of reference resolved MMIs.

A computer program product for use with a computer, the computer program product comprising a computer usable medium having a computer readable program code embodied therein for resolving cross-modal references in user inputs to a data processing system, the user inputs being entered through at least one input modality, the computer program code performing:

• generating a set of multimodal interpretations (MMIs) based on the user inputs collected during a turn, at least one MMI comprising at least one reference, each reference comprising at least one reference variable;
• generating one or more sets of joint MMIs, each set of joint MMIs comprising MMIs of semantically compatible types;
• generating a set of reference resolved MMIs for each set of joint MMIs, wherein the generation of a set of reference resolved MMIs is done by resolving the reference variables of the references of the set of joint MMIs;
• and
• generating an integrated MMI for each set of reference resolved MMIs, wherein the generation of the integrated MMI is done by unifying the set of reference resolved MMIs.