# US 2018/0322631 A1 · High-throughput adaptive sampling for whole-slide histopathology image analysis

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2018/0322631 A1
- Document ID: US20180322631A1
- Status: Published application
- Kind: A1
- Application number: 16/023,361
- Filed: 2018-06-29
- Published: 2018-11-08
- Earliest claimed: 2015-12-03
- Assignees: Case Western Reserve University
- Applicants: Case Western Reserve University
- Inventors: Anant Madabhushi; Angel Alfonso Cruz Roa; Fabio Gonzalez
- Web page: https://patentgrep.com/patent/US20180322631A1

## CPC

- G06T 7/0012: {Biomedical image inspection}
- A61B 10/0041: {Detection of breast cancer (mammography by transillumination A61B5/0091, by X-rays A61B6/502, by ultrasonic means A61B8/0825)}
- G06F 18/2411: based on the proximity to a decision surface, e.g. support vector machines
- G06F 18/2413: based on distances to training or reference patterns
- G06F 18/2415: based on parametric or probabilistic models, e.g. based on likelihood ratio or false acceptance rate versus a false rejection rate
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/08: Learning methods
- G06N 3/09: Supervised learning
- G06N 7/01: Probabilistic graphical models, e.g. probabilistic networks
- G06V 10/44: Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
- G06V 10/454: {Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]}
- G06V 10/50: by performing operations within image blocks; by using histograms, e.g. histogram of oriented gradients [HoG]; by summing image-intensity values; Projection analysis
- G06V 10/764: using classification, e.g. of video objects
- G06V 10/82: using neural networks
- G16H 30/20: for handling medical images, e.g. DICOM, HL7 or PACS
- G16H 50/20: for computer-aided diagnosis, e.g. based on medical expert systems
- G16Z 99/00: Subject matter not provided for in other main groups of this subclass
- G06T 2207/10024: Color image
- G06T 2207/10056: Microscopic image
- G06T 2207/20021: Dividing image into blocks, subimages or windows
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/30024: Cell structures in vitro; Tissue sections in vitro
- G06T 2207/30096: Tumor; Lesion
- G06V 2201/031: of internal organs
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D)
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G16H: HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
- G16Z: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS, NOT OTHERWISE PROVIDED FOR

## Abstract

Methods, apparatus, and other embodiments associated with classifying a region of tissue represented in a digitized whole slide image (WSI) using iterative gradient-based quasi-Monte Carlo (QMC) sampling. One example apparatus includes an image acquisition circuit that acquires a WSI of a region of tissue demonstrating cancerous pathology, an adaptive sampling circuit that selects a subset of tiles from the WSI using an iterative QMC Sobol sequence sampling approach, an invasiveness circuit that determines a probability of a presence of invasive pathology in a member of the subset of tiles, a probability map circuit that generates an invasiveness probability map based on the probability, a probability gradient circuit that generates a gradient image based on the invasiveness probability map, and a classification circuit that classifies the region of tissue based on the probability map. A prognosis or treatment plan may be provided based on the classification of the WSI.

## Claims

### Claim 1

A non-transitory computer-readable storage medium storing computer executable instructions that when executed by a computer control the computer to perform a method for detecting cancerous pathology in a region of tissue, the method comprising:
accessing an image of a region of tissue demonstrating cancerous pathology, where the image comprises a set of tiles;
generating a tile sampling selection, where the tile sampling selection defines a sampling subset of the set of tiles, where the number of tiles in the sampling subset is N, where N is less than the number of tiles in the set of tiles, N being an integer;
upon determining that a termination condition has not been met:
extracting the sampling subset from the image based on the tile sampling selection;
providing the sampling subset to an automated classifier;
receiving, from the automated classifier, a prediction probability based, at least in part, on the sampling subset, where the prediction probability indicates the probability of invasive pathology at a location in the image occupied by a tile;
constructing a probability map of the region of tissue based on the prediction probability or the sampling subset;
computing a gradient of the probability map; and
updating the tile sampling selection based on the gradient of the probability map;
upon determining that the termination condition has been met:
constructing a final probability map based, at least in part, on the probability map; and
controlling a computer aided diagnosis (CADx) system to generate a classification of the region of tissue based, at least in part, on the final probability map.

### Claim 2 (depends on 1)

The non-transitory computer-readable storage medium of claim 1, where the image of the region of tissue is a digitized whole slide image (WSI) of a region of tissue demonstrating estrogen receptor-positive (ER+) BCa.

### Claim 3 (depends on 2)

The non-transitory computer-readable storage medium of claim 2, where the WSI is a red-green-blue (RGB) color model WSI, where the WSI has dimensions of 80,000 pixels by 80,000 pixels.

### Claim 4 (depends on 3)

The non-transitory computer-readable storage medium of claim 3, where a tile has dimensions of 101 pixels by 101 pixels.

### Claim 5 (depends on 1)

The non-transitory computer-readable storage medium of claim 1, where generating the tile sampling selection comprises:
selecting the sampling subset of the set of tiles using a regular sampling approach, a uniform random sampling approach, or a quasi-Monte Carlo sampling approach.

### Claim 6 (depends on 5)

The non-transitory computer-readable storage medium of claim 5, where the quasi-Monte Carlo sampling approach employs a deterministic sequence.

### Claim 7 (depends on 6)

The non-transitory computer-readable storage medium of claim 6, where the deterministic sequence has a low discrepancy.

### Claim 8 (depends on 7)

The non-transitory computer-readable storage medium of claim 7, where the deterministic sequence measures a uniformity of a distribution of finite point sets.

### Claim 9 (depends on 8)

The non-transitory computer-readable storage medium of claim 8, where the deterministic sequence is a Sobol sequence or a Halton sequence.

### Claim 10 (depends on 1)

The non-transitory computer-readable storage medium of claim 1, where the termination condition comprises a number of iterations, a steady state condition, a processing time limit, a minimum epsilon value for an average magnitude gradient of the probability map, or a minimum approximation error between a first probability map and a second, different probability map.

### Claim 11 (depends on 10)

The non-transitory computer-readable storage medium of claim 10, where the number of iterations is twenty.

### Claim 12 (depends on 1)

The non-transitory computer-readable storage medium of claim 1, where the prediction probability represents the probability of invasive pathology at the location in the image occupied by a tile.

### Claim 13 (depends on 12)

The non-transitory computer-readable storage medium of claim 12, where constructing the probability map based on the prediction probability and the set of tiles comprises cubically interpolating the prediction probability to a plurality of tiles in the image.

### Claim 14 (depends on 13)

The non-transitory computer-readable storage medium of claim 13, where computing the gradient of the probability map comprises:
calculating a gradient image based on an X direction gradient and a Y direction gradient, where the X direction gradient represents a directional change of the probability map in the X direction, and where the Y direction gradient represents a directional change of the probability map in the Y direction; and
computing a gradient magnitude image based, at least in part, on the gradient image or the X direction gradient and the Y direction gradient.

### Claim 15 (depends on 14)

The non-transitory computer-readable storage medium of claim 14, where updating the tile sampling selection based on the gradient of the probability map comprises:
selecting 2N sample tiles from the set of tiles using a quasi-Monte Carlo sampling approach or a random sampling approach; and
generating an updated sampling selection by determining the N tiles that have the highest gradient magnitude from the 2N sample tiles, where the updated sampling selection comprises the N tiles that have the highest gradient magnitude from the 2N sample tiles.

### Claim 16 (depends on 15)

The non-transitory computer-readable storage medium of claim 15, where determining the N tiles from the 2N sample tiles that have the highest gradient magnitude comprises:
generating a ranked set of sample tiles based, at least in part, on the gradient magnitude image; and
selecting the N highest ranked sample tiles from the ranked set of sample tiles.

### Claim 17 (depends on 1)

The non-transitory computer-readable storage medium of claim 1, where the automated classifier is a 2-layer convolutional neural network (CNN) comprising:
a first layer comprising a 256 unit convolutional layer and a 256 unit pooling layer;
a second layer comprising a 256 unit fully-connected layer; and
a classification layer, where the classification layer is a softmax classifier, where the classification layer produces a first output activated by a logistic regression function and a second output activated by the logistic regression function.

### Claim 18 (depends on 17)

The non-transitory computer-readable storage medium of claim 17, where the convolutional layer generates a feature map by applying a 2 dimensional (2D) convolution of an input image with an 8 pixel by 8 pixel kernel.

### Claim 19 (depends on 18)

The non-transitory computer-readable storage medium of claim 18, where the pooling layer applies a spatial L2 pooling function without overlapping to the feature map, where the pooling layer applies a 2 pixel by 2 pixel pooling kernel to the feature map.

### Claim 20 (depends on 1)

The non-transitory computer-readable storage medium of claim 1, the method further comprising training the automated classifier.

### Claim 21 (depends on 20)

The non-transitory computer-readable storage medium of claim 20, where training the automated classifier comprises training a CNN model by:
accessing a set of digital WSIs of tissue demonstrating invasive pathology and non-invasive pathology, where a member of the set of digital WSIs comprises a set of tiles, where a member of the set of digital WSIs is an RGB color model WSI;
accessing a training set of tiles, where the training set includes a tile sampled from a region of invasive BCa tissue, and a tile sampled from a region of non-invasive tissue;
converting the training set of tiles from RGB color space to YUV color space;
normalizing the training set of tiles to a mean of zero and a variance of one; and
minimizing a softmax loss function using a stochastic gradient descent.

### Claim 22

A non-transitory computer-readable storage medium storing computer executable instructions that when executed by a computer control the computer to perform a method, the method comprising:
accessing a radiological image of a region of tissue demonstrating invasive pathology, where the radiological image comprises a set of tiles;
selecting a sample set of tiles from the set of tiles using a gradient-based quasi-Monte Carlo sampling approach;
providing the sample set of tiles to an automated classifier;
receiving, from the automated classifier, a classification of the sample set of tiles;
generating an invasiveness probability map based, at least in part, on the classification;
computing a gradient image based, at least in part, on the invasiveness probability map; and
providing an invasiveness classification and localization based on the invasiveness probability map and the gradient image.

### Claim 23

An apparatus for detecting cancerous pathology in a region of tissue, comprising:
a processor;
a memory;
an input/output interface;
a set of circuits; and
an interface to connect the processor, the memory, the input/output interface and the set of circuits, where the set of circuits includes:
an image acquisition circuit that acquires an image of a region of tissue demonstrating cancerous pathology, where the image comprises a set of tiles;
an adaptive sampling circuit that selects a sampling subset of tiles from the image where the sampling subset of tiles comprises N tiles, where N is less that the number of tiles in the set of tiles, N being an integer;
an invasiveness circuit that determines a probability of a presence of invasive pathology in a member of the sampling subset of tiles;
a probability map circuit that generates an invasiveness probability map based, at least in part, on the probability;
a probability gradient circuit that generates a gradient image based, at least in part, on the invasiveness probability map, where the gradient image identifies directional changes of the invasiveness probability map; and
a classification circuit that classifies the region of tissue based, at least in part, on the probability map.

### Claim 24 (depends on 23)

The apparatus of claim 23, where the adaptive sampling circuit selects the sampling subset of tiles using an iterative quasi-Monte Carlo (QMC) Sobol sequence sampling approach or an iterative QMC Halton sequence sampling approach.

### Claim 25 (depends on 23)

The apparatus of claim 23, where the adaptive sampling circuit selects the sampling subset of tiles using a uniform random sampling approach, or a gradient-based random sampling approach.

### Claim 26 (depends on 23)

The apparatus of claim 23, where the adaptive sampling circuit selects an updated subset of tiles from the image based on the gradient image, where the adaptive sampling circuit generates a set of 2N sample tiles using an iterative random sampling approach or an iterative quasi-Monte Carlo (QMC) sampling approach, where a member of the set of 2N sample tiles has a gradient magnitude, where the adaptive sampling circuit ranks the set of 2N sample tiles according to a gradient magnitude associated with a member of the set of 2N sample tiles, and where the adaptive sampling circuit selects the N sample tiles that have the highest gradient magnitudes from the set of 2N sample tiles according to the gradient magnitude.

### Claim 27 (depends on 23)

The apparatus of claim 23, where the invasiveness circuit determines the probability using a two-layer convolutional neural network (CNN) classifier trained using a stochastic gradient descent approach.

### Claim 28 (depends on 27)

The apparatus of claim 27, where the CNN classifier employs a (CS256-FC256) architecture comprising a softmax classification layer.

### Claim 29 (depends on 23)

The apparatus of claim 23, where the probability gradient circuit generates the gradient image based on a gradient in the X direction and a gradient in the Y direction, where the gradient in the X direction identifies a change in the probability in the X direction, and the gradient in the Y direction identifies a change in the probability in the Y direction, where a high gradient magnitude corresponds to a heterogeneous spatial distribution of invasive tissue or non-invasive tissue, and where a low gradient magnitude corresponds with a homogeneous distribution of invasive tissue or non-invasive tissue.

### Claim 30 (depends on 23)

The apparatus of claim 23, where the probability map circuit generates the invasiveness probability map by extending the probability of the presence of invasive pathology in a member of the subset of tiles to member of the set of tiles that is not a member of the subset of tiles using cubic interpolation.

### Claim 31 (depends on 23)

The apparatus of claim 23, where the image is a digitized whole slide image (WSI) of a region of tissue demonstrating estrogen receptor-positive (ER+) BCa.