Method and system for integrating multimodal interpretations

A method for integrating multimodal interpretations (MMIs) generated from user inputs to a data processing system, the user inputs being entered through at least one input modality, the method comprising:

• segmenting the user inputs by collecting sets of multimodal interpretations (MMIs) corresponding to the user inputs for a user turn, at least one MMI comprising at least one semantic relationship function the semantic relationship function denoting a semantic relationship between values determined from the user inputs entered through one or more of the at least one input modality, the semantic relationship function comprising at least one semantic operator;
• semantically classifying the collected MMIs to generate sets of joint MMIs, wherein each set of joint MMIs comprises MMIs of semantically compatible types;
• generating one or more sets of reference resolved MMIs by resolving the reference variables in the sets of joint MMIs, wherein resolving the reference variables refers to replacing each reference variable with a resolved value;
• and
• integrating each set of reference resolved MMIs to generate a corresponding set of integrated MMIs by using semantic operator based fusion.

A data processing system for integrating multimodal interpretations (MMIs) generated from user inputs to the data processing system, the user inputs being entered through at least one input modality, the data processing system generating MMIs based on each user input, the data processing system comprising:

• a segmentation module, the segmentation module collecting a set of MMIs corresponding to the user inputs for a user turn, at least one MMI comprising at least one semantic relationship function the semantic relationship function denoting a semantic relationship between values determined from the user inputs entered through one or more of the at least one input modality, the semantic relationship function comprising at least one semantic operator;
• a semantic classifier, the semantic classifier semantically classifying the set of MMIs to generate sets of joint MMIs, wherein each set of joint MMIs comprises MMIs of semantically compatible types;
• a reference resolution module, the reference resolution module generating one or more sets of reference resolved MMIs by resolving the reference variables in the sets of joint MMIs, wherein resolving the reference variables refers to replacing each reference variable with a resolved value;
• and
• an integrator module, the integrator module integrating each set of reference resolved MMIs to generate a corresponding set of integrated MMIs by using semantic operator based fusion.

An electronic equipment that integrates multimodal interpretations (MMIs) generated from user inputs to a data processing system, the user inputs being entered through at least one input modality, the equipment comprising:

• means for generating a set of MMIs based on the user inputs for a user turn, at least one MMI comprising at least one semantic relationship function, the semantic relationship function denoting a semantic relationship between values determined from the user inputs entered through one or more of the at least one input modality, the semantic relationship function comprising at least one semantic operator;
• means for generating one or more sets of joint MMIs by using the generated set of MMIs, each set of joint MMIs comprising one or more MMIs of semantically compatible types;
• means for resolving reference values within each set of joint MMIs to generate corresponding sets of reference resolved MMIs;
• and
• means for generating an integrated MMI for each set of reference resolved MMIs by using semantic operator based fusion.

A computer program product for use with a computer, the computer program product comprising a computer usable medium having a computer readable program code embodied therein for integrating multimodal interpretations (MMIs) generated from user inputs to a data processing system, the user inputs being entered through at least one input modality, the computer program code including functions for:

• generating a set of MMIs based on the user inputs for a user turn, at least one MMI comprising at least one semantic relationship function, the semantic relationship function denoting a semantic relationship between values determined from the user inputs entered through one or more of the at least one input modality, the semantic relationship function comprising at least one semantic operator;
• generating one or more sets of joint MMIs by using the generated set of MMIs, each set of joint MMIs comprising one or more MMIs of semantically compatible types;
• resolving reference values within each set of joint MMIs to generate corresponding sets of reference resolved MMIs;
• and
• generating an integrated MMI for each set of reference resolved MMIs by using semantic operator based fusion.

A method for integrating multimodal interpretations (MMIs) generated from user inputs to a data processing system, the user inputs being entered through at least one input modality, the method comprising:

• generating a set of MMIs based on the user inputs for a user turn;
• generating one or more sets of joint MMIs by using the generated set of MMIs, each set of joint MMIs comprising one or more MMIs of semantically compatible types;
• resolving reference values within each set of joint MMIs to generate corresponding sets of reference resolved MMIs;
• and
• generating an integrated MMI of an aggregate type for each set of reference resolved MMIs when all MMIs in a corresponding set of reference resolved MMIs are of a same concept type and wherein all features in the set of reference resolved MMIs have non-empty values.