Multi-task multi-modal machine learning system

A computer implemented method comprising:

• receiving a request to perform a machine learning task on an input of a first modality, wherein the machine learning task comprises a machine learning task from a particular machine learning domain that transforms inputs of the first modality to outputs of a second modality;
• selecting an input modality neural network that corresponds to the first modality from a plurality of input modality neural networks, wherein the selected input modality neural network is configured to map data inputs of the first modality to mapped data inputs of a unified representation space;
• processing the input of the first modality using the selected input modality neural network to generate a mapped input of the unified representation space;
• processing the mapped input of the unified representation space an encoder neural network and a decoder neural network to generate a decoder output, the decoder output representing a representation of an output of the machine learning task in the unified representation space;
• selecting an output modality neural network that corresponds to a second modality from a plurality of output modality neural networks, wherein the selected output modality neural network is configured to map decoder outputs of the unified representation space to mapped data outputs of the second modality;
• and
• processing the decoder output using the selected output modality neural network to generate data representing an output of the second modality of the machine learning task.