Machine-learned hormone status prediction from image analysis

A non-transitory computer-readable storage medium storing instructions that, when executed by a processor, cause the processor to perform operations comprising:

• accessing a test haematoxylin and eosin (H&E) stain image of a test tissue sample;
• partitioning the test H&E stain image into a plurality of image tiles;
• sampling a subset of image tiles from the plurality of image tiles for the test H&E stain image;
• generating a tile feature vector for each image tile of the sampled subset of image tiles;
• generating an aggregate feature vector for the test H&E stain image by aggregating the tile feature vectors of the image tiles in the sampled subset;
• and
• predicting a hormone receptor status by applying a machine-learned prediction model to the aggregate feature vector for the test H&E stain image, wherein the machine-learned prediction model is trained using a first set of H&E stain images from a first set of tissue samples having a positive hormone receptor status and a second set of H&E stain images from a second set of tissue samples having a negative hormone receptor status.

A method for training one or more machine-learned models configured to predict hormone receptor status comprising:

• accessing a first set of H&E stain images from a first set of tissue samples having a positive hormone receptor status and a second set of H&E stain images from a second set of tissue samples having a negative hormone receptor status;
• partitioning each H&E stain image in the first set of H&E stain images and the second set of H&E stain images into a plurality of image tiles;
• for each partitioned H&E stain image, sampling a subset of image tiles from the plurality of image tiles;
• initializing a tile featurization model configured to generate a feature vector for an image tile, an attention model configured to generate an aggregate feature vector for a H&E stain image by aggregating feature vectors for a sampled subset of image tiles from the H&E stain image, and a prediction model configured to generate a prediction of hormone receptor status based on an aggregate feature vector;
• and
• training the tile featurization model, the attention model, and the prediction model using the first set of H&E stain images and the second set of H&E stain images.