Multimodal machine learning based clinical predictor

A system comprising:

• one or more processors programmed and configured to execute a set of instructions, the set of instructions comprising:
• receiving first molecular sequencing data of a patient;
• receiving first biopsy image data of the patient;
• receiving second molecular sequencing data and second biopsy image data of a plurality of patients;
• executing a multi-stage machine learning model with the received first biopsy image data, wherein executing the multi-stage machine learning model comprises:
• an identification of at least one of a shape or a boundary from pixel data of biopsy image data;
• recognition operation using the identified shape or boundary comprising at least one of:
• a tissue segmentation operation to identify a tissue and tissue type, a cell extraction operation to identify a cell, or a cell component extraction operation to identify a cell component;
• and
• a computing of biopsy feature data based on the at least one of:
• a tissue and tissue type identified by the tissue segmentation operation, a count of a cell extracted by the cell extraction operation, a density of the cell extracted by the cell extraction operation, or a cell component identified by the cell component extraction operation;
• wherein the multi-stage machine learning model is trained with the second biopsy image data of a plurality of patients;
• training a multi-modal machine learning model with the second molecular sequencing data and with biopsy image feature data computed by applying the multi-stage machine learning model to the second biopsy-image data, to output a clinical prediction of a treatment, wherein the multi-modal machine learning model comprises one or more of a random forest model, a support vector machine model, and/or a neural network model;
• and
• executing the multi-modal machine learning model with input of the first molecular data and with input of biopsy feature data computed by applying the multi-stage machine learning model to the first biopsy image data, to generate an output of a clinical prediction of a treatment.

A non-transitory computer readable medium storing instructions that, when executed by one or more processors, cause the one or more processor to execute the instructions, the instructions comprising:

• receiving first molecular sequencing data of a patient;
• receiving first biopsy image data of the patient;
• receiving second molecular sequencing data and second biopsy image data of a plurality of patients;
• executing a multi-stage machine learning model with the received first biopsy image data, wherein executing the multi-stage machine learning model comprises:
• an identification of at least one of a shape or a boundary from pixel data of biopsy image data;
• recognition operation using the identified shape or boundary comprising at least one of:
• a tissue segmentation operation to identify a tissue and tissue type, a cell extraction operation to identify a cell, or a cell component extraction operation to identify a cell component;
• and
• a computing of biopsy feature data based on the at least one of:
• a tissue and tissue type identified by the tissue segmentation operation, a count of a cell extracted by the cell extraction operation, a density of the cell extracted by the cell extraction operation, or a cell component identified by the cell component extraction operation;
• wherein the multi-stage machine learning model is trained with the second biopsy image data of a plurality of patients;
• training a multi-modal machine learning model with the second molecular sequencing data and with biopsy image feature data computed by applying the multi-stage machine learning model to the second biopsy-image data, to output a clinical prediction of a treatment, wherein the multi-modal machine learning model comprises one or more of a random forest model, a support vector machine model, and/or a neural network model;
• and
• executing the multi-modal machine learning model with input of the first molecular data and with input of biopsy feature data computed by applying the multi-stage machine learning model to the first biopsy image data, to generate an output of a clinical prediction of a treatment.