# US 12,762,694 B2 · Systems and methods for image classification using visual dictionaries > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,762,694 B2 - Document ID: US12762694B2 - Status: Granted - Kind: B2 - Application number: 18/126,594 - Filed: 2023-03-27 - Published: 2023-09-21 - Granted: 2026-10-13 - Earliest claimed: 2019-03-05 - Assignees: Memorial Sloan Kettering Cancer Center - Applicants: Memorial Sloan Kettering Cancer Center - Inventors: Thomas J. Fuchs; Hassan Muhammad - Web page: https://patentgrep.com/patent/US12762694B2 ## CPC - G06T 7/0012: {Biomedical image inspection} - G06F 18/214: Generating training patterns; Bootstrap methods, e.g. bagging or boosting - G06F 18/23: Clustering techniques - G06F 18/2431: Multiple classes - G06N 3/0455: Auto-encoder networks; Encoder-decoder networks - G06V 10/25: Determination of region of interest [ROI] or a volume of interest [VOI] - G06V 10/50: by performing operations within image blocks; by using histograms, e.g. histogram of oriented gradients [HoG]; by summing image-intensity values; Projection analysis - G06V 10/762: using clustering, e.g. of similar faces in social networks - G06V 10/7715: {Feature extraction, e.g. by transforming the feature space, e.g. multi-dimensional scaling [MDS]; Mappings, e.g. subspace methods} - G06V 10/774: Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting - G06V 10/82: using neural networks - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract Presented herein are systems and methods of clustering images using encoder-decoder models. A computing system may identify tiles derived from an image. Each tile may have a first dimension. The computing system may apply an image reconstruction model to the tiles. The image reconstruction model may include an encoder block having a first set of weights to generate embedding representations corresponding to the tiles. Each embedding representation may have a second dimension lower than the first dimension. The image reconstruction model may include a decoder block having a second set of weights to generate reconstructed tiles corresponding to the embedding representations. The computing system may apply a clustering model comprising a feature space to the embedding representations to classify each tile to one of a plurality of conditions. ## Claims ### Claim 1 A method of classifying biomedical images into subtypes of cancer, comprising: identifying, by a computing system, a first biomedical image of a first tissue section having a first region of interest; generating, by the computing system, a first plurality of tiles from a portion of the first biomedical image; feeding, by the computing system, the first plurality of tiles to an image reconstruction system comprising: an encoder block having a first set of weights to output a plurality of embedding representations, wherein the encoder block is established using a (i) second biomedical image derived from a second tissue section having a second region of interest and (ii) an annotation identifying the second region of interest as associated with one of a plurality of conditions; a decoder block having a second set of weights to generate a plurality of reconstructed tiles corresponding to the plurality of embedding representations; applying, by the computing system, a clustering model to the plurality of embedding representations, wherein the clustering model comprises a feature space defining a plurality of centroids, the plurality of centroids including one centroid for each condition of the plurality of conditions, and wherein each centroid represents a respective one of the plurality of conditions identified by the annotation, each centroid being generated from training embedding representations output by the encoder block for tiles from the second biomedical image that are associated, based on the annotation, with the respective condition; classifying, by the computing system, the first biomedical image as a subtype of cancer by: identifying, for each embedding representation of the plurality of embedding representations, a corresponding point within the feature space, calculating a distance metric between each corresponding point and each centroid of the plurality of centroids, assigning, for each tile of the first plurality of tiles, the condition represented by the centroid having a smallest distance metric to the corresponding point, and determining the classification of the first biomedical image as the subtype of cancer based at least in part on the assigned condition for each tile; and storing, and/or displaying, by the computing system, an output identifying the classification of the first biomedical image to the subtype of cancer. ### Claim 2 (depends on 1) The method of claim 1, further comprising generating, by the computing system, a survival function over time for a subject from which the first tissue section is obtained, based on the classification of the first biomedical image to the subtype of cancer. ### Claim 3 (depends on 1) The method of claim 1, further comprising determining, by the computing system, a survival probability for a subject from which the first tissue section is obtained based on the classification. ### Claim 4 (depends on 3) The method of claim 3, further comprising assigning, by the computing system, the subject from which the first tissue section is obtained into one of a plurality of risk stratification groups, based on the survival probability determined using the classification. ### Claim 5 (depends on 1) The method of claim 1, wherein each condition of the plurality of conditions specifies a classification for a region of interest. ### Claim 6 (depends on 1) The method of claim 1, wherein identifying the first biomedical image further comprises receiving, via an imaging device, the first biomedical image corresponding to at least one tile in a whole slide image (WSI) of the first tissue section stained for histopathological analysis. ### Claim 7 (depends on 1) The method of claim 1, wherein the clustering model comprises a feature space defining a plurality of regions, each of the plurality of regions corresponding to at least one of a cellular morphology or a structural morphology for a respective subtype of cancer. ### Claim 8 A system for classifying biomedical images into subtypes of cancer, comprising: one or more processors coupled with memory storing operations, the operations comprising: identifying, by a computing system, a first biomedical image of a first tissue section having a first region of interest; generating, by the computing system, a first plurality of tiles from a portion of the first biomedical image; feeding, by the computing system, the first plurality of tiles to an image reconstruction system comprising: an encoder block having a first set of weights to output a plurality of embedding representations, wherein the encoder block is established using a (i) second biomedical image derived from a second tissue section having a second region of interest and (ii) an annotation identifying the second region of interest as associated with one of a plurality of conditions; a decoder block having a second set of weights to generate a plurality of reconstructed tiles corresponding to the plurality of embedding representations; applying, by the computing system, a clustering model to the plurality of embedding representations, wherein the clustering model comprises a feature space defining a plurality of centroids, wherein the clustering model comprises a feature space defining a plurality of centroids, the plurality of centroids including one centroid for each condition of the plurality of conditions, and wherein each centroid represents a respective one of the plurality of conditions identified by the annotation, each centroid being generated from training embedding representations output by the encoder block for tiles from the second biomedical image that are associated, based on the annotation, with the respective condition; classifying, by the computing system, the first biomedical image as a subtype of cancer by: identifying, for each embedding representation of the plurality of embedding representations, a corresponding point within the feature space, calculating a distance metric between each corresponding point and each centroid of the plurality of centroids, assigning, for each tile of the first plurality of tiles, the condition represented by the centroid having a smallest distance metric to the corresponding point, and determining the classification of the first biomedical image as the subtype of cancer based at least in part on the assigned condition for each tile; and storing, and/or displaying, by the computing system, an output identifying the classification of the first biomedical image to the subtype of cancer. ### Claim 9 (depends on 8) The system of claim 8, wherein the one or more processors are further configured to generate a survival function over time for a subject from which the first tissue section is obtained, based on the classification of the first biomedical image to the subtype of cancer. ### Claim 10 (depends on 8) The system of claim 8, wherein the one or more processors are further configured to determine a survival probability for a subject from which the first tissue section is obtained based on the classification. ### Claim 11 (depends on 10) The system of claim 10, wherein the one or more processors are further configured to assign the subject from which the first tissue section is obtained into one of a plurality of risk stratification groups, based on the survival probability determined using the classification. ### Claim 12 (depends on 8) The system of claim 8, wherein each condition of the plurality of conditions specifies a classification for a region of interest. ### Claim 13 (depends on 8) The system of claim 8, wherein the one or more processors are further configured to receive, via an imaging device, the first biomedical image corresponding to at least one tile in a whole slide image (WSI) of the first tissue section stained for histopathological analysis. ### Claim 14 A non-transitory computer readable medium storing instructions that, when executed by a processor, cause the processor to perform a method of classifying biomedical images into subtypes of cancer, comprising: identifying, by a computing system, a first biomedical image of a first tissue section having a first region of interest; generating, by the computing system, a first plurality of tiles from a portion of the first biomedical image; feeding, by the computing system, the first plurality of tiles to an image reconstruction system comprising: an encoder block having a first set of weights to output a plurality of embedding representations, wherein the encoder block is established using a (i) second biomedical image derived from a second tissue section having a second region of interest and (ii) an annotation identifying the second region of interest as associated with one of a plurality of conditions; a decoder block having a second set of weights to generate a plurality of reconstructed tiles corresponding to the plurality of embedding representations; applying, by the computing system, a clustering model to the plurality of embedding representations, wherein the clustering model comprises a feature space defining a plurality of centroids, wherein the clustering model comprises a feature space defining a plurality of centroids, the plurality of centroids including one centroid for each condition of the plurality of conditions, and wherein each centroid represents a respective one of the plurality of conditions identified by the annotation, each centroid being generated from training embedding representations output by the encoder block for tiles from the second biomedical image that are associated, based on the annotation, with the respective condition; classifying, by the computing system, the first biomedical image as a subtype of cancer by: identifying, for each embedding representation of the plurality of embedding representations, a corresponding point within the feature space, calculating a distance metric between each corresponding point and each centroid of the plurality of centroids, assigning, for each tile of the first plurality of tiles, the condition represented by the centroid having a smallest distance metric to the corresponding point, and determining the classification of the first biomedical image as the subtype of cancer based at least in part on the assigned condition for each tile; and storing, and/or displaying, by the computing system, an output identifying the classification of the first biomedical image to the subtype of cancer. ### Claim 15 (depends on 14) The non-transitory computer readable medium of claim 14, further comprising generating, by the computing system, a survival function over time for a subject from which the first tissue section is obtained, based on the classification of the first biomedical image to the subtype of cancer. ### Claim 16 (depends on 14) The non-transitory computer readable medium of claim 14, further comprising determining, by the computing system, a survival probability for a subject from which the first tissue section is obtained based on the classification. ### Claim 17 (depends on 16) The non-transitory computer readable medium of claim 16, further comprising assigning, the subject from which the first tissue section is obtained into one of a plurality of risk stratification groups, based on the survival probability determined using the classification. ### Claim 18 (depends on 14) The non-transitory computer readable medium of claim 14, wherein each condition of the plurality of conditions specifies a classification for a region of interest. ### Claim 19 (depends on 14) The non-transitory computer readable medium of claim 14, wherein the clustering model comprises a feature space defining a plurality of regions, each of the plurality of regions corresponding to at least one of a cellular morphology or a structural morphology for a subtype of cancer. ### Claim 20 (depends on 14) The non-transitory computer readable medium of claim 14, wherein identifying the first biomedical image further comprises receiving, via an imaging device, the first biomedical image corresponding to at least one tile in a whole slide image (WSI) of the first tissue section stained for histopathological analysis.