Multimodal data inference

A computer program product for generating a machine learning model for a multimodal data inference task, the computer program product comprising a computer readable storage medium having program instructions embodied therein, the program instructions being executable by a computing system to cause the computing system, for each sample in a training dataset of multimodal data samples, to:

• encode the sample to produce a compressed vector representation of the sample in a k-dimensional latent space;
• perturb features of the sample to identify, for each of a plurality of dimensions of the k-dimensional latent space, a set of active features, perturbation of each of which produces more than a threshold change in the compressed vector representation in that dimension;
• generate an active feature map based on the set of active features using each of the plurality of dimensions of the k-dimensional latent space;
• create a node map having a plurality of sides, each side of the node map including a plurality of nodes from a respective one of the different modalities of the training dataset, wherein the nodes represent respective features of the sample;
• for each of the plurality of dimensions of the k-dimensional latent space and based on the active feature map, create a sub-sample graph defining edges interconnecting the nodes of the node map for that dimension, wherein the edges indicate the active features for each dimension;
• generate a sample graph by superimposing each of the sub-sample graphs for each of the plurality of dimensions;
• and
• use the sample graph to train a graph neural network model to perform the multimodal data inference task.

A multimodal data inference system comprising:

• a sample graph generator adapted to, for each sample in a training dataset including a plurality of data samples having different modalities:
• encode an input multimodal data the sample to produce a compressed vector representation of the sample in a k-dimensional latent space,
• perturb features of the sample to identify, for each of a plurality of dimensions of the k-dimensional latent space, a set of active features, perturbation of each of which produces more than a threshold change in the compressed vector representation in that dimension,
• generate an active feature map based on the set of active features using each of the plurality of dimensions of the k-dimensional latent space;
• create a node map having a plurality of sides, each side of the node map including a plurality of nodes from a respective one of the different modalities of the training dataset, wherein the nodes represent respective features of the sample;
• for each of the plurality of dimensions of the k-dimensional latent space and based on the active feature map, create a sub-sample graph defining edges interconnecting the nodes of the node map for that dimension, wherein the edges indicate the active features for each dimension;
• generate a sample graph by superimposing each of the sub-sample graphs for each of the plurality of dimensions;
• and
• a graph neural network model, pretrained to perform a multimodal data inference task using sample graphs generated by the sample graph generator for respective samples in a training dataset of multimodal data samples;
• wherein the sample graph generator is adapted to supply the sample graph for the input multimodal data sample to the graph neural network model to obtain an inference output for the sample.