Back to results
US 11,900,703 B2Granted
Systems and methods for automated tagging of digital histology slides
AssigneeHistoWiz, Inc.
InventorsKe Cheng, Matthew Wilder, Weijian Li
Life of this patent
Estimating the term…
Abstract
Provided herein are methods and systems for performing an automated tagging of features in digital micrographs representing slides with tissue samples. Automated tagging of features may include automated entry of metadata associated with whole slide images or regions of the whole slide images.
Claims26
A method of automatic multi-tagging of whole slide histopathology images comprising:
- a) receiving a digital whole slide image;
- b) sampling a plurality of non-background patches from the whole slide image at a plurality of magnifications;
- c) applying a machine learning model to predict multiple slide-level tags from the plurality of non-background patches, wherein the machine learning model comprises:
- i. a first module configured to extract a visual feature representation of each patch;
- ii. a second module configured to modulate patch features by providing the extracted visual feature representations as inputs to a first attention aggregation mechanism performing operations comprising:
- mapping each visual feature representation into different attention-weighted features through multi-head attentions and outputting a weighted patch feature comprising an aggregation of the visual feature representations and the attention-weighted features;
- wherein the machine learning model operates in a multi-scale mode utilizing the patches at the plurality of magnifications to identify both patch features and magnification levels most discriminant for the multi-tagging;
- iii. a third module configured to construct slide-level tags identifying tag classifications by providing the weighted patch features to a plurality of second attention aggregation mechanisms, one per output head, each performing operations comprising:
- mapping each weighted patch feature into different attention-weighted features through multi-head attentions, and outputting a slide-level tag comprising an aggregation of the attention-weighted features;
- and
- d) associating the multiple slide-level tags with the digital whole slide image.
8 more elements
A system comprising:
- at least one processor, a memory, and instructions executable by the at least one processor to create a whole slide histopathology image multi-tagging application comprising:
- a) a software module configured to receive a digital whole slide image and sample a plurality of non-background patches from the whole slide image at a plurality of magnifications;
- b) a machine learning model to predict multiple slide-level tags from the plurality of non-background patches, wherein the machine learning model comprises:
- i. a visual feature extraction module configured to extract a visual feature representation of each patch;
- ii. a patch transformation module configured to modulate patch features by providing the extracted visual feature representations as inputs to a first attention aggregation mechanism performing operations comprising:
- mapping each visual feature representation into different attention-weighted features through multi-head attentions and outputting a weighted patch feature comprising an aggregation of the visual feature representations and the attention-weighted features;
- wherein the machine learning model operates in a multi-scale mode utilizing the patches at the plurality of magnifications to identify both patch features and magnification levels most discriminant for the multi-tagging;
- iii. a multi-tag attention module configured to construct slide-level tags identifying tag classifications by providing the weighted patch features to a plurality of second attention aggregation mechanisms, one per output head, each performing operations comprising:
- mapping each weighted patch feature into different attention-weighted features through multi-head attentions, and outputting a slide-level tag comprising an aggregation of the attention-weighted features;
- and
- c) a software module configured to associate the multiple slide-level tags with the digital whole slide image.
8 more elements
A non-transitory computer-readable storage media encoded with a computer program including instructions executable by a processor to create a whole slide histopathology image multi-tagging application comprising:
- a) an intake module configured to receive a digital whole slide image and sample a plurality of non-background patches from the whole slide image at a plurality of magnifications;
- b) a machine learning model to predict multiple slide-level tags from the plurality of non-background patches, wherein the machine learning model comprises:
- i. a visual feature extraction module configured to extract a visual feature representation of each patch;
- ii. a patch transformation module configured to modulate patch features by providing the extracted visual feature representations as inputs to a first attention aggregation mechanism performing operations comprising:
- mapping each visual feature representation into different attention-weighted features through multi-head attentions and outputting a weighted patch feature comprising an aggregation of the visual feature representations and the attention-weighted features;
- wherein the machine learning model operates in a multi-scale mode utilizing the patches at the plurality of magnifications to identify both patch features and magnification levels most discriminant for the multi-tagging;
- iii. a multi-tag attention module configured to construct slide-level tags identifying tag classifications by providing the weighted patch features to a plurality of second attention aggregation mechanisms, one per output head, each performing operations comprising:
- mapping each weighted patch feature into different attention-weighted features through multi-head attentions, and outputting a slide-level tag comprising an aggregation of the attention-weighted features;
- and
- c) a tagging module configured to associate the multiple slide-level tags with the digital whole slide image.
7 more elements
Dependencies are inferred from the claim text.