Modality-specific and modality-generic latent representations

A method for processing multi-modal data, the method comprising:

• inputting a first set of features and a second set of features into a fusion model;
• obtaining as output from the fusion model:
• at least one of:
• a first set of modality-specific features associated with a first modality;
• or
• a second set of modality-specific features associated with a second modality, wherein the first set of modality-specific features includes one or more first types of features that are distinct from one or more second types of features included in the second set of modality-specific features;
• and
• a set of modality-generic features associated with both the first modality and the second modality;
• and
• obtaining, as output from one or more subsequent processing modules, a result based on the one or more of the first set of modality-specific features, the second set of modality-specific features, or the set of modality-generic feature.

One or more non-transitory computer-readable media comprising executable instructions that, when executed by one or more processors, cause the one or more processors to perform operations comprising:

• inputting a first set of features and a second set of features into a fusion model;
• obtaining, as output from the fusion model:
• at least one of:
• a first set of modality-specific features associated with a first modality;
• or
• a second set of modality-specific features associated with a second modality, wherein the first set of modality-specific features includes one or more first types of features that are distinct from one or more second types of features included in the second set of modality-specific features;
• and
• a set of modality-generic features associated with both the first modality and the second modality;
• and
• obtaining, as output from one or more subsequent processing modules, a result based on the one or more of the first set of modality-specific features, the second set of modality-specific features, or the set of modality-generic features.