Channel fusion for vision-language representation learning

A computer system for multi-modal processing, the computer system comprising:

• one or more processors;
• and
• one or more non-transitory computer-readable media that collectively store a machine-learned multi-modal model, the machine-learned multi-modal model comprising one or more multi-modal fusion layers, each multi-modal fusion layer configured to:
• obtain a plurality of first embedding tokens associated with a first data modality;
• obtain a plurality of second embedding tokens associated with a second data modality that is different from the first data modality;
• perform cross-attention using the plurality of first embedding tokens as queries and the plurality of second embedding tokens as keys and values to generate a plurality of second-to-first intermediate tokens;
• concatenate the plurality of second-to-first intermediate tokens with the plurality of first embedding tokens in a channel-wise fashion to generate a plurality of second-to-first compound tokens;
• perform cross-attention using the plurality of second embedding tokens as queries and the plurality of first embedding tokens as keys and values to generate a plurality of first-to-second intermediate tokens;
• and
• concatenate the plurality of first-to-second intermediate tokens with the plurality of second embedding tokens in a channel-wise fashion to generate a plurality of first-to-second compound tokens.

A computer system for multi-modal processing, the computer system comprising:

• one or more processors;
• and
• one or more non-transitory computer-readable media that collectively store a machine-learned multi-modal model, the machine-learned multi-modal model comprising one or more multi-modal fusion layers, each multi-modal fusion layer configured to:
• obtain a plurality of first embedding tokens associated with a first data modality;
• obtain a plurality of second embedding tokens associated with a second data modality that is different from the first data modality;
• perform cross-attention using the plurality of first embedding tokens as queries and the plurality of second embedding tokens as keys and values to generate a plurality of second-to-first intermediate tokens;
• and
• concatenate the plurality of second-to-first intermediate tokens with the plurality of first embedding tokens in a channel-wise fashion to generate a plurality of second-to-first compound tokens;
• wherein the machine-learned multi-modal model further comprises a decoder configured to process the plurality of second-to-first compound tokens to generate a prediction for a multi-modal task.

A computer system for multi-modal processing, the computer system comprising:

• one or more processors;
• and
• one or more non-transitory computer-readable media that collectively store a machine-learned multi-modal model, the machine-learned multi-modal model comprising one or more multi-modal fusion layers, each multi-modal fusion layer configured to:
• obtain a plurality of first embedding tokens associated with a first data modality;
• obtain a plurality of second embedding tokens associated with a second data modality that is different from the first data modality;
• perform cross-attention using the plurality of first embedding tokens as queries and the plurality of second embedding tokens as keys and values to generate a plurality of second-to-first intermediate tokens;
• and
• concatenate the plurality of second-to-first intermediate tokens with the plurality of first embedding tokens in a channel-wise fashion to generate a plurality of second-to-first compound tokens;
• wherein the machine-learned multi-modal model further comprises one or more self-attention layers configured to perform self-attention on the plurality of second-to-first compound tokens to generate a set of output compound tokens;
• and
• wherein the machine-learned multi-modal model further comprises a decoder configured to process the set of output compound tokens to generate a prediction for a multi-modal task.