# US 2025/0166824 A1 · System and Method for Annotating Pathology Images to Predict Outcome

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2025/0166824 A1
- Document ID: US20250166824A1
- Status: Published application
- Kind: A1
- Application number: 18/841,323
- Filed: 2023-02-24
- Published: 2025-05-22
- Earliest claimed: 2022-02-24
- Assignees: The Johns Hopkins University
- Applicants: The Johns Hopkins University
- Inventors: Janis M. Taube; Julie Stein; Vinay Pulim; Tricia Ruth Cottrell
- Web page: https://patentgrep.com/patent/US20250166824A1

## CPC

- G16H 30/40: for processing medical images, e.g. editing
- G16H 50/20: for computer-aided diagnosis, e.g. based on medical expert systems
- G06T 7/0012: {Biomedical image inspection}
- G06V 10/24: Aligning, centring, orientation detection or correction of the image
- G06V 10/26: Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion
- G06V 10/764: using classification, e.g. of video objects
- G06V 10/82: using neural networks
- G06V 20/695: {Preprocessing, e.g. image segmentation}
- G06V 20/698: {Matching; Classification}
- G06V 20/70: Labelling scene content, e.g. deriving syntactic or semantic representations
- G16H 10/60: for patient-specific data, e.g. for electronic patient records
- G16H 50/50: for simulation or modelling of medical disorders
- G16H 50/70: for mining of medical data, e.g. analysing previous cases of other patients
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/096: Transfer learning
- G06T 2207/10056: Microscopic image
- G06T 2207/20021: Dividing image into blocks, subimages or windows
- G06T 2207/20081: Training; Learning
- G06T 2207/30024: Cell structures in vitro; Tissue sections in vitro
- G06T 2207/30096: Tumor; Lesion
- G06V 10/776: Validation; Performance evaluation
- G06V 2201/03: Recognition of patterns in medical or anatomical images
- G16H 10/40: for data related to laboratory analysis, e.g. patient specimen analysis
- G16H 15/00: ICT specially adapted for medical reports, e.g. generation or transmission thereof
- G16H 30/20: for handling medical images, e.g. DICOM, HL7 or PACS
- G16H: HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

Methods, systems, apparatus, and computer programs, for predicting clinical outcome of a patient. In one aspect, a method includes obtaining a pathology image associated with the patient; processing the pathology image including: determining a region of interest of the pathology image; and segmenting the pathology image into a plurality of tiles, wherein the plurality of tiles has a uniform size; providing the processed pathology image as an input to a machine learning model configured to annotate the pathology image and predict the clinical outcome of the patient based on the annotated pathology image; obtaining, as an output of the machine learning model, the annotated pathology image associated with the patient, wherein the annotated pathology image includes annotations for different classes of tissues including a tumor regression; and determining, based on the annotated pathology image, a metric indicative of the clinical outcome of the patient.

## Claims

### Claim 1

A computer-implemented method for predicting a clinical outcome of a patient, the method comprising:
obtaining a pathology image associated with the patient;
processing the pathology image including:
determining a region of interest of the pathology image; and
segmenting the pathology image into a plurality of tiles, wherein each of the plurality of tiles has a uniform size;
providing the processed pathology image as an input to a machine learning model configured to annotate the pathology image;
obtaining, as an output of the machine learning model, the annotated pathology image associated with the patient, wherein the annotated pathology image includes annotations for different classes of tissues including a tumor regression; and
determining, based on the annotated pathology image, a metric indicative of the clinical outcome of the patient.

### Claim 2 (depends on 1)

The computer-implemented method of claim 1, wherein the clinical outcome of the patient comprises responsiveness to one or more treatment regimens, survival, disease recurrence, and disease progression.

### Claim 3 (depends on 1)

The computer-implemented method of claim 1, wherein the pathology image comprises a histopathology slide image.

### Claim 4 (depends on 1)

The computer-implemented method of claim 1,
wherein the machine learning model has been trained on a plurality of training data items, and
wherein each training data item includes an annotated pathology image with tissue classes.

### Claim 5 (depends on 1)

The computer-implemented method of claim 1, further comprising:
generating rendering data, when rendered by a user device, causes the user device to display a user interface that displays the annotated pathology image associated with the patient and the predicted clinical outcome of the patient.

### Claim 6 (depends on 5)

The computer-implemented method of claim 5, wherein the user interface comprises a user selectable element to upload the pathology image for applying the machine learning model.

### Claim 7 (depends on 1)

The computer-implemented method of claim 1, wherein the different classes of tissues comprise tumor, necrosis, area of tumor regression, and normal background, wherein the normal background indicates an uninvolved tissue.

### Claim 8

(canceled)

### Claim 9

(canceled)

### Claim 10 (depends on 1)

The computer-implemented method of claim 1, wherein processing the pathology image further comprises:
normalizing color of the pathology image based on color of a set of pathology images; and
applying image transformations to the pathology image, wherein the image transformations include rotation, shift, flip, zoom, affine transformation, and adjusting brightness and contrast.

### Claim 11

(canceled)

### Claim 12

(canceled)

### Claim 13

A system for predicting a clinical outcome of a patient comprising:
one or more computers; and
one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations, the operations comprising:
obtaining a pathology image associated with the patient;
processing the pathology image including:
determining a region of interest of the pathology image; and
segmenting the pathology image into a plurality of tiles, wherein each of the plurality of tiles has a uniform size;
providing the processed pathology image as an input to a machine learning model configured to annotate the pathology image;
obtaining, as an output of the machine learning model, the annotated pathology image associated with the patient, wherein the annotated pathology image includes annotations for different classes of tissues including a tumor regression; and
determining, based on the annotated pathology image, a metric indicative of the clinical outcome of the patient.

### Claim 14 (depends on 13)

The system of claim 13, wherein the clinical outcome of the patient comprises responsiveness to one or more treatment regimens, survival, disease recurrence, and disease progression.

### Claim 15 (depends on 13)

The system of claim 13, wherein the pathology image comprises a histopathology slide image.

### Claim 16 (depends on 13)

The system of claim 13,
wherein the machine learning model has been trained on a plurality of training data items, and
wherein each training data item includes an annotated pathology image with tissue classes.

### Claim 17 (depends on 13)

The system of claim 13, the operations further comprising:
generating rendering data, when rendered by a user device, causes the user device to display a user interface that displays the annotated pathology image associated with the patient and the predicted clinical outcome of the patient.

### Claim 18 (depends on 17)

The system of claim 17, wherein the user interface comprises a user selectable element to upload the pathology image for applying the machine learning model.

### Claim 19 (depends on 13)

The system of claim 13, wherein the different classes of tissues comprise tumor, necrosis, area of tumor regression, and normal background, wherein the normal background indicates an uninvolved tissue.

### Claim 20 (depends on 13)

The system of claim 13, wherein the metric indicative of the clinical outcome of the patient comprises a percent residual viable tumor, wherein the percent residual viable tumor is computed based on areas of the different classes of tissues in the pathology image.

### Claim 21 (depends on 13)

The system of claim 13, wherein the metric indicative of the clinical outcome of the patient comprises a percent regression and a percent necrosis in the pathology image.

### Claim 22 (depends on 13)

The system of claim 13, wherein processing the pathology image further comprises:
normalizing color of the pathology image based on color of a set of pathology images; and
applying image transformations to the pathology image, wherein the image transformations include rotation, shift, flip, zoom, affine transformation, and adjusting brightness and contrast.

### Claim 23

One or more computer-readable storage medium storing instructions that, when executed by one or more computers, cause the one or more computers to perform operations for predicting a clinical outcome of a patient, the operations comprising:
obtaining a pathology image associated with the patient;
processing the pathology image including:
determining a region of interest of the pathology image; and
segmenting the pathology image into a plurality of tiles, wherein each of the plurality of tiles has a uniform size;
providing the processed pathology image as an input to a machine learning model configured to annotate the pathology image;
obtaining, as an output of the machine learning model, the annotated pathology image associated with the patient, wherein the annotated pathology image includes annotations for different classes of tissues including a tumor regression; and
determining, based on the annotated pathology image, a metric indicative of the clinical outcome of the patient.

### Claim 24 (depends on 13)

The computer-readable storage medium of claim 13, wherein processing the pathology image further comprises:
normalizing color of the pathology image based on color of a set of pathology images; and
applying image transformations to the pathology image, wherein the image transformations include rotation, shift, flip, zoom, affine transformation, and adjusting brightness and contrast.
