Multimodal deep learning with boosted trees

A system having a memory, computer readable instructions, and one or more processors for executing the computer readable instructions, the computer readable instructions controlling the one or more processors to perform operations comprising:

• training a plurality of unimodal teacher models, wherein each unimodal teacher model of the plurality of unimodal teacher models is trained using training data from a unique modality of a plurality of modalities;
• for each of the unimodal teacher models, training a respective student encoder of a plurality of student encoders using a knowledge distillation whereby one or more features for each respective student encoder are forced to a same feature of the respective unimodal teacher model;
• feeding a concatenation of outputs from the plurality of student encoders to train a fusion neural network of the multimodal neural network;
• receiving data from the plurality of modalities;
• and
• generating a prediction from an output layer of the trained fusion neural network.

A computer program product comprising a computer readable storage medium having program instructions embodied therewith, the program instructions executable by one or more processors to cause the one or more processors to perform operations comprising:

• training a plurality of unimodal teacher models, wherein each unimodal teacher model of the plurality of unimodal teacher models is trained using training data from a unique modality of a plurality of modalities;
• for each of the unimodal teacher models, training a respective student encoder of a plurality of student encoders using a knowledge distillation whereby one or more features for each respective student encoder are forced to a same feature of the respective unimodal teacher model;
• feeding a concatenation of outputs from the plurality of student encoders to train a fusion neural network of the multimodal neural network;
• receiving data from the plurality of modalities;
• and
• generating a prediction from an output layer of the trained fusion neural network.