System and method for multimodal prediction of patient outcomes

A method of predicting overall survivability of a patient by a prediction system based on machine learning, the method comprising:

• receiving, by the prediction system, a plurality of input modalities corresponding to the patient, the input modalities comprising:
• a first modality comprising histology hematoxylin and eosin (H&E) image data;
• a second modality comprising genetic sequencing data;
• a third modality comprising clinical data;
• a fourth modality comprising radiology magnetic resonance imaging (MRI) data;
• and
• a fifth modality comprising immunohistochemistry (IHC) image data;
• generating, by the prediction system, a plurality of intermediate features based on the plurality of input modalities, each input modality of the plurality of input modalities corresponding to one or more features of the plurality of intermediate features;
• and
• determining, by the prediction system, a survivability score corresponding to an overall survivability of the patient based on a fusion of the plurality of intermediate features.

A prediction system comprising:

• a first convolutional neural network configured to receive a histology hematoxylin and eosin (H&E) image and to generate one or more first intermediate features;
• a first feed forward neural network configured to receive genetic sequencing data and to generate one or more second intermediate features;
• a second feed forward neural network configured to receive clinical data and to generate one or more third intermediate features;
• a second convolutional neural network configured to receive radiology magnetic resonance imaging (MRI) data and to generate one or more fourth intermediate features;
• a fusion circuit configured to combine the first, second, third, and fourth intermediate features to form a fusion of intermediate features via attention-gated tensor fusion;
• and
• a fusion layer neural network configured to receive the fusion of intermediate features and to generate a survivability score corresponding to an overall survivability of a patient.