Systems and methods for image classification using visual dictionaries

A system for classifying biomedical images into subtypes of cancer, comprising:

• one or more processors coupled with memory storing operations, the operations comprising:
• identifying, by a computing system, a first biomedical image of a first tissue section having a first region of interest;
• generating, by the computing system, a first plurality of tiles from a portion of the first biomedical image;
• feeding, by the computing system, the first plurality of tiles to an image reconstruction system comprising:
• an encoder block having a first set of weights to output a plurality of embedding representations, wherein the encoder block is established using a (i) second biomedical image derived from a second tissue section having a second region of interest and (ii) an annotation identifying the second region of interest as associated with one of a plurality of conditions;
• a decoder block having a second set of weights to generate a plurality of reconstructed tiles corresponding to the plurality of embedding representations;
• applying, by the computing system, a clustering model to the plurality of embedding representations, wherein the clustering model comprises a feature space defining a plurality of centroids, wherein the clustering model comprises a feature space defining a plurality of centroids, the plurality of centroids including one centroid for each condition of the plurality of conditions, and wherein each centroid represents a respective one of the plurality of conditions identified by the annotation, each centroid being generated from training embedding representations output by the encoder block for tiles from the second biomedical image that are associated, based on the annotation, with the respective condition;
• classifying, by the computing system, the first biomedical image as a subtype of cancer by:
• identifying, for each embedding representation of the plurality of embedding representations, a corresponding point within the feature space,
• calculating a distance metric between each corresponding point and each centroid of the plurality of centroids,
• assigning, for each tile of the first plurality of tiles, the condition represented by the centroid having a smallest distance metric to the corresponding point, and
• determining the classification of the first biomedical image as the subtype of cancer based at least in part on the assigned condition for each tile;
• and
• storing, and/or displaying, by the computing system, an output identifying the classification of the first biomedical image to the subtype of cancer.

A non-transitory computer readable medium storing instructions that, when executed by a processor, cause the processor to perform a method of classifying biomedical images into subtypes of cancer, comprising:

• identifying, by a computing system, a first biomedical image of a first tissue section having a first region of interest;
• generating, by the computing system, a first plurality of tiles from a portion of the first biomedical image;
• feeding, by the computing system, the first plurality of tiles to an image reconstruction system comprising:
• an encoder block having a first set of weights to output a plurality of embedding representations, wherein the encoder block is established using a (i) second biomedical image derived from a second tissue section having a second region of interest and (ii) an annotation identifying the second region of interest as associated with one of a plurality of conditions;
• a decoder block having a second set of weights to generate a plurality of reconstructed tiles corresponding to the plurality of embedding representations;
• applying, by the computing system, a clustering model to the plurality of embedding representations, wherein the clustering model comprises a feature space defining a plurality of centroids, wherein the clustering model comprises a feature space defining a plurality of centroids, the plurality of centroids including one centroid for each condition of the plurality of conditions, and wherein each centroid represents a respective one of the plurality of conditions identified by the annotation, each centroid being generated from training embedding representations output by the encoder block for tiles from the second biomedical image that are associated, based on the annotation, with the respective condition;
• classifying, by the computing system, the first biomedical image as a subtype of cancer by:
• identifying, for each embedding representation of the plurality of embedding representations, a corresponding point within the feature space,
• calculating a distance metric between each corresponding point and each centroid of the plurality of centroids,
• assigning, for each tile of the first plurality of tiles, the condition represented by the centroid having a smallest distance metric to the corresponding point, and
• determining the classification of the first biomedical image as the subtype of cancer based at least in part on the assigned condition for each tile;
• and
• storing, and/or displaying, by the computing system, an output identifying the classification of the first biomedical image to the subtype of cancer.