Multi-modal fusion techniques considering inter-modality correlations and computer model uncertainty

A computer program product comprising a computer readable storage medium having a computer readable program stored therein, wherein the computer readable program, when executed on a data processing system, causes the data processing system to be configured to provide a joint multimodal fusion computer model architecture which operates to:

• receive prediction output data from a machine learning (ML) computer model set comprising a plurality of different subsets of ML computer models, each subset of ML computer models in the ML computer model set operating on input data of a different modality than input data of other subsets of ML computer models in the ML computer model set, and each subset of ML computer models generating a plurality of different prediction output in the prediction output data;
• fuse prediction outputs in the plurality of different prediction output data by executing an uncertainty and correlation weighted (UCW) joint multimodal fusion operation on the prediction output data to generate a fused output comprising a multimodal prediction output data, wherein the uncertainty and correlation weight(UCW) joint multimodal fusion operation comprises:
• generating an uncertainty estimation for each subset of ML computer models;
• generating a pairwise modality correlation matrix data structure where each entry in the pairwise modality correlation matrix data structure specifies at least one value indicating a degree of correlation between prediction outputs generated by pairings of subsets of ML computer models corresponding to the different modalities;
• and generate different weights for the prediction outputs corresponding to different subsets of ML computer models based on a function of the uncertainty estimations for each subset of ML computer models and the pairwise modality correlation matrix data structure;
• applying the different weights to different ones of prediction outputs from the different subsets of ML computer models operating on input data of the different modalities;
• and output the fused output comprising the multimodal prediction output data to a computing device, wherein the different weights are determined based on the function of the uncertainty estimation for each of the different subsets of ML computer models and an estimate of the pairwise modality correlation matrix data structure between the different modalities.

An apparatus comprising:

• at least one processor;
• and at least one memory coupled to the at least one processor, wherein the at least one memory comprises instructions which, when executed by the at least one processor, cause the at least one processor to be configured to provide a joint multimodal fusion computer model architecture which operates to:
• receive prediction output data from a machine learning (ML) computer model set comprising a plurality of different subsets of ML computer models, each subset of ML computer models in the ML computer model set operating on input data of a different modality than input data of other subsets of ML computer models in the ML computer model set,
• and each subset of ML computer models generating a plurality of different prediction output in the prediction output data;
• fuse prediction outputs in the plurality of different prediction output data by executing an uncertainty and correlation weighted (UCW) joint multimodal fusion operation on the prediction output data to generate a fused output comprising multimodal prediction output data, wherein the uncertainty and correlation weight(UCW) joint multimodal fusion operation comprises:
• generating an uncertainty estimation for each subset of ML computer models;
• generating a pairwise modality correlation matrix data structure where each entry in the pairwise modality correlation matrix data structure specifies at least one value indicating a degree of correlation between prediction outputs generated by pairings of subsets of ML computer models corresponding to the different modalities;
• and generate different weights for the prediction outputs corresponding to different subsets of ML computer models based on a function of the uncertainty estimations for each subset of ML computer models and the pairwise modality correlation matrix data structure;
• applying the different weights to different ones of prediction outputs from the different subsets of ML computer models operating on input data of the different modalities;
• and output the fused output comprising the multimodal prediction output data to a computing device, wherein the different weights are determined based on [an} the function of the uncertainty estimation for each of the different subsets of ML computer models and an estimate of the pairwise modality correlation matrix data structure between the different modalities.