Multimodal artificial neural network method and system for precision oncology

A system comprising one or more processors configured to perform the operations of:

• receiving whole slide images (WSI) each comprising an image file of multiple pixels;
• extracting image features from the received WSI images using one or more feature extraction architectures comprising one or more of a convolutional neural network, a vision transformer, or other foundation model;
• generating clinical vector embeddings by inputting numerical clinical data into a clinical data encoder;
• generating slide vector embeddings by inputting the extracted image features via a slide encoder;
• generating gene vector embeddings by inputting into a gene encoder scout gene expressions;
• inputting into a contrastive module comprising a machine learning model, the generated clinical vector embeddings, the slide vector embeddings and the gene vector embeddings, and determining a treatment score and/or a response to therapy score;
• and
• providing for display an indication of the determined score.