Weighted deep fusion architecture

A computer program product for fusing data for multi-modal classifications for a plurality of modality inputs, the computer program product comprising:

• one or more non-transitory computer-readable storage media and program instructions stored on the one or more non-transitory computer-readable storage media capable of performing a method, the method comprising:
• receiving a request indicative of the modality inputs to be selected;
• performing an embeddings level fusion operation to concatenate features from the modality inputs;
• performing a multi-modal discriminative feature level fusion operation that integrates feature representations learned by applying different network structures on the modality inputs;
• determining weights of the concatenated features and the feature representations based on a measure of the concatenated features and the feature representations indicative of affecting a final prediction performance;
• generating fused features for the modality inputs based on the concatenated features, the feature representations, and the weights;
• generating a response to the request based on the fused features;
• and
• transmitting the response.

A computer system for fusing data for multi-modal classifications for a plurality of modality inputs, the computer system comprising:

• one or more computer processors, one or more computer-readable storage media, and program instructions stored on the one or more of the computer-readable storage media for execution by at least one of the one or more processors capable of performing a method, the method comprising:
• receiving a request indicative of the modality inputs to be selected;
• performing an embeddings level fusion operation to concatenate features from the modality inputs;
• performing a multi-modal discriminative feature level fusion operation that integrates feature representations learned by applying different network structures on the modality inputs;
• determining weights of the concatenated features and the feature representations based on a measure of the concatenated features and the feature representations indicative of affecting a final prediction performance;
• generating fused features for the modality inputs based on the concatenated features, the feature representations, and the weights;
• generating a response to the request based on the fused features;
• and
• transmitting the response.