# US 11,908,139 B1 · Systems and methods for training a statistical model to predict tissue characteristics for a pathology image

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,908,139 B1
- Document ID: US11908139B1
- Status: Granted
- Kind: B1
- Application number: 18/352,097
- Filed: 2023-07-13
- Granted: 2024-02-20
- Earliest claimed: 2017-06-06
- Assignees: PathAI, Inc.
- Applicants: PathAI, Inc.
- Inventors: Andrew H. Beck; Aditya Khosla
- Web page: https://patentgrep.com/patent/US11908139B1

## CPC

- G16H 50/30: for calculating health indices; for individual health risk assessment
- G06F 16/55: Clustering; Classification
- G06F 16/5854: {using shape and object relationship}
- G06F 16/5866: {using information manually generated, e.g. tags, keywords, comments, manually generated location and time information}
- G06F 18/214: Generating training patterns; Bootstrap methods, e.g. bagging or boosting
- G06F 18/2193: {based on specific statistical tests}
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/08: Learning methods
- G06N 3/09: Supervised learning
- G06N 3/091: Active learning
- G06N 5/046: Forward inferencing; Production systems
- G06T 7/0012: {Biomedical image inspection}
- G06T 7/0014: {using an image reference approach}
- G06T 7/194: involving foreground-background segmentation
- G06V 10/764: using classification, e.g. of video objects
- G06V 10/82: using neural networks
- G16H 10/20: for electronic clinical trials or questionnaires
- G16H 30/40: for processing medical images, e.g. editing
- G16H 50/20: for computer-aided diagnosis, e.g. based on medical expert systems
- G16H 50/50: for simulation or modelling of medical disorders
- G16H 50/70: for mining of medical data, e.g. analysing previous cases of other patients
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G06N 3/048: Activation functions
- G06N 5/01: Dynamic search techniques; Heuristics; Dynamic trees; Branch-and-bound
- G06N 20/10: using kernel methods, e.g. support vector machines [SVM]
- G06N 20/20: Ensemble learning
- G06T 2207/10056: Microscopic image
- G06T 2207/20021: Dividing image into blocks, subimages or windows
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/20104: Interactive definition of region of interest [ROI]
- G06T 2207/30004: Biomedical image processing
- G06T 2207/30024: Cell structures in vitro; Tissue sections in vitro
- G06T 2207/30096: Tumor; Lesion
- G06V 2201/03: Recognition of patterns in medical or anatomical images
- G16H: HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

In some aspects, the described systems and methods provide for a method for training a statistical model to predict tissue characteristics for a pathology image. The method includes accessing annotated pathology images. Each of the images includes an annotation describing a tissue characteristic category for a portion of the image. A set of training patches and a corresponding set of annotations are defined using an annotated pathology image. Each of the training patches in the set includes values obtained from a respective subset of pixels in the annotated pathology image and is associated with a corresponding patch annotation determined based on an annotation associated with the respective subset of pixels. The statistical model is trained based on the set of training patches and the corresponding set of patch annotations. The trained statistical model is stored on at least one storage device.

## Claims

### Claim 1

A method for predicting an entity of interest for a pathology image, the method comprising, by one or more processors:
accessing a pathology image of a patient;
processing, using a trained deep learning model trained on a plurality of annotated images, the pathology image to determine tissue and cell characteristics for the pathology image;
extracting values for one or more features based on the tissue and cell characteristics for the pathology image;
predicting an entity of interest using the values for the one or more features, wherein the entity of interest includes one or more of patient response, tumor molecular characteristics, and patient clinical outcomes; and
outputting the predicted entity of interest associated with the patient for storing on at least one storage device.

### Claim 2 (depends on 1)

The method of claim 1, wherein processing, using the trained deep learning model, the pathology to determine the tissue and cell characteristics comprises:
using a first deep learning model to identify cells characteristics for the pathology image; and
using a second deep learning model to identify tissue characteristics for the pathology image.

### Claim 3 (depends on 1)

The method of claim 1, wherein the pathology image is a H&E whole slide image.

### Claim 4 (depends on 1)

The method of claim 1, wherein the predicted entity of interest comprises response to drug or therapy associated with the patient.

### Claim 5 (depends on 1)

The method of claim 1, wherein the predicted entity of interest comprises prognosis of cancer for the patient.

### Claim 6 (depends on 1)

The method of claim 1, wherein the predicted entity of interest comprises tumor molecular characteristics including genomic markers.

### Claim 7 (depends on 1)

The method of claim 1, wherein the values for one or more features comprise spatial features related to spatial distribution of cells, heterogeneity, and texture.

### Claim 8 (depends on 1)

The method of claim 1, wherein predicting the entity of interest is performed additionally using clinical metadata associated with the patient.

### Claim 9 (depends on 1)

The method of claim 1, wherein the plurality of annotated pathology images are annotated by human pathologists.

### Claim 10 (depends on 1)

The method of claim 1, further comprising displaying one or more annotations overlaid on the pathology image, the one or more annotations describing the determined tissue and cell characteristics for the pathology image.

### Claim 11 (depends on 1)

The method of claim 1, wherein outputting the predicted entity of interest comprises displaying a value indicating a measurement of patient response.

### Claim 12

A system for predicting an entity of interest for a pathology image, the system comprising:
at least one computer hardware processor; and
at least one non-transitory computer-readable storage medium storing processor-executable instructions that, when executed by the at least one computer hardware processor, cause the at least one computer hardware processor to perform:
accessing a pathology image of a patient;
processing, using a trained deep learning model trained on a plurality of annotated images, the pathology image to determine tissue and cell characteristics for the pathology image;
extracting values for one or more features based on the tissue and cell characteristics for the pathology image;
predicting an entity of interest using the values for the one or more features, wherein the entity of interest includes one or more of patient response, tumor molecular characteristics, and patient clinical outcomes; and
outputting the predicted entity of interest associated with the patient for storing on at least one storage device.

### Claim 13 (depends on 12)

The system of claim 12, wherein processing, using the trained deep learning model, the pathology to determine the tissue and cell characteristics comprises:
using a first deep learning model to identify cells characteristics for the pathology image; and
using a second deep learning model to identify tissue characteristics for the pathology image.

### Claim 14 (depends on 12)

The system of claim 12, wherein the predicted entity of interest comprises response to drug or therapy associated with the patient.

### Claim 15 (depends on 12)

The system of claim 12, wherein the predicted entity of interest comprises prognosis of cancer for the patient.

### Claim 16 (depends on 12)

The system of claim 12, wherein the predicted entity of interest comprises tumor molecular characteristics including genomic markers.

### Claim 17 (depends on 12)

The system of claim 12, wherein the values for one or more features comprise spatial features related to spatial distribution of cells, heterogeneity, and texture.

### Claim 18 (depends on 12)

The system of claim 12, wherein predicting the entity of interest is performed additionally using clinical metadata associated with the patient.

### Claim 19 (depends on 12)

The system of claim 12, wherein the at least one computer hardware processor is further configured to perform:
causing to display one or more annotations overlaid on the pathology image, the one or more annotations describing the determined tissue and cell characteristics for the pathology image.

### Claim 20 (depends on 12)

The system of claim 12, wherein outputting the predicted entity of interest comprises causing to display a value indicating a measurement of patient response.

### Claim 21

A non-transitory computer-readable medium containing instructions that, when executed, cause at least one computer hardware processor to perform:
accessing a pathology image of a patient;
processing, using a trained deep learning model trained on a plurality of annotated images, the pathology image to determine tissue and cell characteristics for the pathology image;
extracting values for one or more features based on the tissue and cell characteristics for the pathology image;
predicting an entity of interest using the values for the one or more features, wherein the entity of interest includes one or more of patient response, tumor molecular characteristics, and patient clinical outcomes; and
outputting the predicted entity of interest associated with the patient for storing on at least one storage device.

### Claim 22 (depends on 21)

The non-transitory computer-readable medium of claim 21, wherein processing, using the trained deep learning model, the pathology to determine the tissue and cell characteristics comprises:
using a first deep learning model to identify cells characteristics for the pathology image; and
using a second deep learning model to identify tissue characteristics for the pathology image.

### Claim 23 (depends on 21)

The non-transitory computer-readable medium of claim 21, wherein the predicted entity of interest comprises response to drug or therapy associated with the patient.

### Claim 24 (depends on 21)

The non-transitory computer-readable medium of claim 21, wherein the predicted entity of interest comprises prognosis of cancer for the patient.

### Claim 25 (depends on 21)

The non-transitory computer-readable medium of claim 21, wherein the predicted entity of interest comprises tumor molecular characteristics including genomic markers.

### Claim 26 (depends on 21)

The non-transitory computer-readable medium of claim 21, wherein the plurality of annotated pathology images are annotated by human pathologists.