# US 12,002,262 B2 · Digital histopathology and microdissection > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,002,262 B2 - Document ID: US12002262B2 - Status: Granted - Kind: B2 - Application number: 18/088,487 - Filed: 2022-12-23 - Published: 2023-04-27 - Granted: 2024-06-04 - Earliest claimed: 2016-10-21 - Assignees: NantOmics, LLC - Applicants: NantOmics, LLC - Inventors: Bing Song; Gregory Chu - Web page: https://patentgrep.com/patent/US12002262B2 ## CPC - G06V 10/82: using neural networks - G06F 18/21: Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation - G06F 18/2193: {based on specific statistical tests} - G06F 18/23213: with fixed number of clusters, e.g. K-means clustering - G06F 18/2411: based on the proximity to a decision surface, e.g. support vector machines - G06F 18/24137: {Distances to cluster centroïds} - G06F 18/24147: {Distances to closest patterns, e.g. nearest neighbour classification} - G06F 18/24155: {Bayesian classification} - G06F 18/285: {Selection of pattern recognition techniques, e.g. of classifiers in a multi-classifier system} - G06F 18/29: {Graphical models, e.g. Bayesian networks} - G06N 3/045: Combinations of networks - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/09: Supervised learning - G06N 7/01: Probabilistic graphical models, e.g. probabilistic networks - G06T 7/0012: {Biomedical image inspection} - G06T 7/11: Region-based segmentation - G06T 7/187: involving region growing; involving region merging; involving connected component labelling - G06V 10/454: {Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]} - G06V 10/50: by performing operations within image blocks; by using histograms, e.g. histogram of oriented gradients [HoG]; by summing image-intensity values; Projection analysis - G06V 10/764: using classification, e.g. of video objects - G06V 10/7796: {based on specific statistical tests} - G06V 20/695: {Preprocessing, e.g. image segmentation} - G06N 20/00: Machine learning - G06T 2207/10056: Microscopic image - G06T 2207/20021: Dividing image into blocks, subimages or windows - G06T 2207/20084: Artificial neural networks [ANN] - G06T 2207/20156: Automatic seed setting - G06T 2207/30024: Cell structures in vitro; Tissue sections in vitro - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract A computer implemented method of generating at least one shape of a region of interest in a digital image is provided. The method includes obtaining, by an image processing engine, access to a digital tissue image of a biological sample; tiling, by the image processing engine, the digital tissue image into a collection of image patches; identifying, by the image processing engine, a set of target tissue patches from the collection of image patches as a function of pixel content within the collection of image patches; assigning, by the image processing engine, each target tissue patch of the set of target tissue patches an initial class probability score indicating a probability that the target tissue patch falls within a class of interest, the initial class probability score generated by a trained classifier executed on each target tissue patch; generating, by the image processing engine, a first set of tissue region seed patches by identifying target tissue patches having initial class probability scores that satisfy a first seed region criteria, the first set of tissue region seed patches comprising a subset of the set of target tissue patches; generating, by the image processing engine, a second set of tissue region seed patches by identifying target tissue patches having initial class probability scores that satisfy a second seed region criteria, the second set of tissue region seed patches comprising a subset of the set of target tissue patches; calculating, by the image processing engine, a region of interest score for each patch in the second set of tissue region seed patches as a function of initial class probability scores of neighboring patches of the second set of tissue region seed patches and a distance to patches within the first set of issue region seed patches; and generating, by the image processing engine, one or more region of interest shapes by grouping neighboring patches based on their region of interest scores. ## Claims ### Claim 1 A computer implemented method of generating at least one shape of a region of interest in a digital image, the method comprising: identifying, by an image processing engine, a set of target patches from a collection of image patches as a function of pixel content within the collection of image patches, wherein the collection of image patches comprises a tiled digital image of a sample; assigning, by the image processing engine, each target patch of the set of target patches an initial class probability score indicating a probability that the target patch falls within a class of interest, the initial class probability score generated by a trained classifier executed on each target patch; generating, by the image processing engine, a first set of patches by identifying target patches having initial class probability scores that satisfy a first criteria, the first set of patches comprising a subset of the set of target patches; generating, by the image processing engine, a second set of patches by identifying target patches having initial class probability scores that satisfy a second criteria, the second set of patches comprising a subset of the set of target patches; calculating, by the image processing engine, a region-of-interest score for each patch in the second set of patches as a function of initial class probability scores of neighboring patches of the second set of patches and a distance to patches within the first set of patches; and generating, by the image processing engine, one or more region-of-interest shapes by grouping neighboring patches based on their region-of-interest scores. ### Claim 2 (depends on 1) The method of claim 1, wherein the tiled digital image is generated by creating image patches that are of a uniform size and a uniform shape. ### Claim 3 (depends on 2) The method of claim 2, wherein the image patches of uniform size and uniform shape include square patches of less than or equal to 1,000 pixels by 1,000 pixels. ### Claim 4 (depends on 3) The method of claim 3, wherein the square patches are less than or equal to 400 pixels by 400 pixels. ### Claim 5 (depends on 4) The method of claim 4, wherein the square patches are less than or equal to 256 pixels by 256 pixels. ### Claim 6 (depends on 1) The method of claim 1, wherein the tiled digital image is generated by creating image patches that are of non-uniform size and shape. ### Claim 7 (depends on 1) The method of claim 1, wherein the collection of image patches includes non-overlapping patches. ### Claim 8 (depends on 1) The method of claim 1, wherein identifying the set of target patches includes filtering the collection of image patches based on color channels of pixels within the image patches. ### Claim 9 (depends on 8) The method of claim 8, further comprising filtering image patches of the collection of image patches as a function of variance with respect to the color channels. ### Claim 10 (depends on 1) The method of claim 1, wherein the trained classifier includes a trained neural network. ### Claim 11 (depends on 1) The method of claim 1, wherein the trained classifier includes a trained implementation of at least one of the following types of classifiers: support vector machine, softmax, decision tree, random forest, k-nearest neighbor, linear and quadratic discriminant analysis, ridge regression, multilayer perceptron (MLP), Hyper-pipes, Bayes net, k-means clustering and naïve Bayes. ### Claim 12 (depends on 1) The method of claim 1, further comprising causing a computing device to render the one or more region-of-interest shapes on a display. ### Claim 13 (depends on 1) The method of claim 1, wherein the one or more region-of-interest shapes include at least one tissue mask. ### Claim 14 (depends on 13) The method of claim 13, wherein the at least one tissue mask comprises a microdissection mask. ### Claim 15 (depends on 1) The method of claim 1, wherein the class of interest comprises at least one cancer class. ### Claim 16 (depends on 15) The method of claim 15, wherein the at least one cancer class includes one of the following types of cancer: breast cancer, bladder cancer, brain cancer, lung cancer, pancreatic cancer, skin cancer, colorectal cancer, prostate cancer, stomach cancer, liver cancer, cervical cancer, esophageal cancer, leukemia, non-Hodgkin lymphoma, kidney cancer, uterine cancer, bile duct cancer, bone cancer, ovarian cancer, gallbladder cancer, gastrointestinal cancer, oral cancer, throat cancer, ocular cancer, pelvic cancer, spinal cancer, testicular cancer, vaginal cancer, vulvar cancer, and thyroid cancer. ### Claim 17 (depends on 1) The method of claim 1, wherein the class of interest comprises at least one of the following types of tissue: abnormal tissue, benign tissue, malignant tissue, bone tissue, skin tissue, nerve tissue, interstitial tissue, muscle tissue, connective tissue, scar tissue, lymphoid tissue, fat, epithelial tissue, nervous tissue, and blood vessels. ### Claim 18 (depends on 1) The method of claim 1, wherein the tiled digital image comprises a slide image of a sample slide. ### Claim 19 (depends on 18) The method of claim 18, wherein the slide image comprises a digital histopathology image. ### Claim 20 (depends on 1) The method of claim 1, further comprising obtaining access to a database of a priori trained classifiers. ### Claim 21 (depends on 20) The method of claim 20, further comprising selecting the trained classifier from the a priori trained classifiers according to classifier selection criteria defined according to biological sample metadata bound to the digital image. ### Claim 22 (depends on 21) The method of claim 21, wherein the biological sample metadata includes digital information associated with at least one of the following: a tissue type, a tissue donor, a scanner, a stain, a staining technique, an identifier of a preparer, an image size, a sample identifier, a tracking identifier, a version number, a file type, an image date, a symptom, a diagnosis, an identifying information of treating physician, a medical history of the tissue donor, a demographic information of the tissue donor, a medical history of family of the tissue donor, and a species of the tissue donor. ### Claim 23 (depends on 1) The method of claim 1, wherein calculating a region-of-interest score for each patch in the second set of patches as a function of neighboring patches includes calculating a conditional random field (CRF) among the neighboring patches. ### Claim 24 (depends on 1) The method of claim 1, wherein calculating a region-of-interest score for each patch in the second set of patches as a function of neighboring patches includes evaluating nearest neighbors. ### Claim 25 (depends on 1) The method of claim 1, wherein the one or more region-of-interest shapes include shapes composed of patches. ### Claim 26 (depends on 1) The method of claim 1, wherein the one or more region-of-interest shapes comprise shape sub-patch level features. ### Claim 27 (depends on 26) The method of claim 26, wherein the one or more region-of-interest shapes comprise sub-patch level shapes at the cell level. ### Claim 28 (depends on 26) The method of claim 26, wherein generating the one or more region-of-interest shapes includes classifying cells by cell-type within patches in the second set of patches according to a cell-level trained neural network. ### Claim 29 (depends on 27) The method of claim 27, wherein generating the one or more region-of-interest shapes includes delineating a boundary between cells via a trained fully convolutional neural network. ### Claim 30 (depends on 29) The method of claim 29, wherein the trained fully convolutional neural network is trained at a patch level on known cell types within training patches. ### Claim 31 (depends on 1) The method of claim 1, wherein the assigning the initial class probability score of a target patch is performed without consideration of the initial class probability of any other patch of the set of target patches. ### Claim 32 (depends on 1) The method of claim 1, wherein neighboring patches of the second set of patches comprise one or more patches adjacent to at least one patch of the second set of patches. ### Claim 33 (depends on 1) The method of claim 1, further comprising, training the trained classifier by a method comprising: receiving, by a convolutional neural network, a plurality of patches of digital images, the plurality of patches including a plurality of normal patches and a plurality of positive patches; setting, by the convolutional neural network, an input size of one or more fully connected layers of an architecture of the convolutional neural network to a new value, the new value being determined based on a cardinality of types of patches in the plurality of patches; populating, by the convolutional neural network, a distribution of values of parameters of the one or more fully connected layers; adjusting, by the convolutional neural network, the values of the parameters of the one or more fully connected layers; assigning, by the convolutional neural network, an initial class probability score to each of the plurality of patches, the initial class probability score indicating a probability that the target patch falls within a group of normal patches or a group of positive patches; and determining whether first initial class probability scores of the plurality of patches are more accurate than second initial class probability scores of the plurality of patches. ### Claim 34 (depends on 33) The method of claim 33, wherein the plurality of digital images includes a cardinality of normal patches that is substantially similar to a cardinality of positive patches. ### Claim 35 (depends on 33) The method of claim 33, wherein the determining that the first initial class probability scores of the plurality of patches are more accurate than the second initial class probability scores of the plurality of patches comprises determining that a sum of squares of a difference between the first initial class probability scores of the plurality of patches and correct initial class probability scores of the plurality of patches is lower than a sum of squares of a difference between the second initial class probability scores of the plurality of patches and the correct initial class probability scores of the plurality of patches. ### Claim 36 (depends on 35) The method of claim 35, wherein the assignment of the second initial class probability scores of the plurality of patches occurs earlier than the assignment of the first initial class probability scores of the plurality of patches. ### Claim 37 (depends on 33) The method of claim 33, wherein the one or more fully connected layers comprise a softmax layer. ### Claim 38 (depends on 33) The method of claim 33, wherein the types of patches are the normal patches and the positive patches. ### Claim 39 (depends on 33) The method of claim 33, wherein the new value comprises 1×1×2. ### Claim 40 (depends on 33) The method of claim 33, wherein the distribution of values of parameters of the one or more fully connected layers is a Gaussian distribution. ### Claim 41 A computer implemented method of generating at least one shape of a region of interest in a digital image, the method comprising: assigning, by an image processing engine, each target patch of a set of target patches an initial class probability score indicating a probability that the target patch falls within a class of interest, the initial class probability score generated by a trained classifier executed on each target patch, wherein the set of target patches are included in a collection of image patches, the collection of image patches comprising a tiled digital image of a sample; generating, by the image processing engine, a first set of patches by identifying target patches having initial class probability scores that satisfy a first criteria, the first set of patches comprising a subset of the set of target patches; generating, by the image processing engine, a second set of patches by identifying target patches having initial class probability scores that satisfy a second criteria, the second set of patches comprising a subset of the set of target patches; calculating, by the image processing engine, a region-of-interest score for each patch in the second set of patches as a function of initial class probability scores of neighboring patches of the second set of patches and a distance to patches within the first set of patches; and generating, by the image processing engine, one or more region-of-interest shapes by grouping neighboring patches based on their region-of-interest scores. ### Claim 42 A computer implemented method of generating at least one shape of a region of interest in a digital image, the method comprising: obtaining, by an image processing engine, a plurality of features from each patch in a collection of image patches, the plurality of features defining a patch feature vector in a multidimensional feature space including the plurality of features as dimensions, wherein the collection of image patches comprises a tiled digital image of a sample; assigning, by the image processing engine, each patch in the collection of image patches an initial class probability score indicating a probability that the patch falls within a class of interest, the initial class probability score generated by a trained classifier executed on each patch in the collection of image patches using the plurality of features of each patch in the collection of image patches; generating, by the image processing engine, a first set of patches by identifying patches in the collection of image patches having initial class probability scores that satisfy a first criteria; generating, by the image processing engine, a second set of patches by identifying patches in the collection of image patches having initial class probability scores that satisfy a second criteria; and generating, by the image processing engine, one or more region-of-interest shapes as a function of, for each patch in the collection of image patches, its inclusion in the first set of patches and the second set of patches. ### Claim 43 A digital image region of interest generation system, the system comprising: a database configured to store a digital image of a sample; an image processing engine configured to: identify a set of target patches from a collection of image patches as a function of pixel content within the collection of image patches, wherein the collection of image patches comprises a tiled digital image of the sample; assign each target patch of the set of target patches an initial class probability score indicating a probability that the target patch falls within a class of interest, the initial class probability score generated by a trained classifier executed on each target patch; generate a first set of patches by identifying target patches having initial class probability scores that satisfy a first criteria, the first set of patches comprising a subset of the set of target patches; generate a second set of patches by identifying target patches having initial class probability scores that satisfy a second criteria, the second set of patches comprising a subset of the set of target patches; calculate a region-of-interest score for each patch in the second set of patches as a function of initial class probability scores of neighboring patches of the second set of patches and a distance to patches within the first set of patches; and generate one or more region-of-interest shapes by grouping neighboring patches based on their region-of-interest scores. ### Claim 44 A digital image region of interest generation system, the system comprising: a database configured to store a digital image of a sample; an image processing engine configured to: obtain a plurality of features from each patch in a collection of image patches, the plurality of features defining a patch feature vector in a multidimensional feature space including the plurality of features as dimensions, wherein the collection of image patches comprises a tiled digital image of the sample; assign each patch in the collection of image patches an initial class probability score indicating a probability that the patch falls within a class of interest, the initial class probability score generated by a trained classifier executed on each patch in the collection of image patches using the plurality of features of each patch in the collection of image patches; generate a first set of patches by identifying patches in the collection of image patches having initial class probability scores that satisfy a first criteria; generate a second set of patches by identifying patches in the collection of image patches having initial class probability scores that satisfy a second criteria; and generate one or more region-of-interest shapes as a function of, for each patch in the collection of image patches, its inclusion in the first set of patches and the second set of patches.