# US 2025/0166174 A1 · Creating a tumor bud ground truth of h&e stained slides and automated identification of tumor buds using an h&e stained slides

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2025/0166174 A1
- Document ID: US20250166174A1
- Status: Published application
- Kind: A1
- Application number: 18/730,593
- Filed: 2023-01-20
- Published: 2025-05-22
- Earliest claimed: 2022-01-20
- Assignees: Ohio State Innovation Foundation; Wake Forest University Health Sciences
- Applicants: Ohio State Innovation Foundation; Wake Forest University Health Sciences
- Inventors: Metin N. GURCAN; Wendy L. FRANKEL; Wei CHEN; Muhammad Khalid Khan NIAZI
- Web page: https://patentgrep.com/patent/US20250166174A1

## CPC

- G06T 7/0012: {Biomedical image inspection}
- G16H 30/40: for processing medical images, e.g. editing
- G16H 50/20: for computer-aided diagnosis, e.g. based on medical expert systems
- G06T 2207/10024: Color image
- G06T 2207/10056: Microscopic image
- G06T 2207/20036: Morphological image processing
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/30024: Cell structures in vitro; Tissue sections in vitro
- G06T 2207/30096: Tumor; Lesion
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G16H: HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA

## Abstract

Tumor budding (TB) is defined as a cluster of one to four tumor cells at the tumor invasive front. Though promising as a prognostic factor for colorectal cancer, its routine clinical use is hampered by high inter-and intra-observer disagreement on routine H&E staining. Therefore, automated methods are provided that minimize the inter-and intra-observer disagreement for tumor bud detection on images of H&E-stained tissue sections.

## Claims

### Claim 1

A method for identifying tumor buds in an image of a hematoxylin & eosin (H&E) stained section of a tumor, the method comprising:
create a ground truth of H&E stained slides having identified regions of tumor buds and/or identified regions that are not tumor buds;
training an artificial intelligence (AI) algorithm using the dataset of H&E stained slides;
analyze the image of the H&E stained section of the tumor using the trained AI algorithm to determine a presence or absence of tumor buds on the H&E stained slide.

### Claim 2 (depends on 1)

The method of claim 1, wherein creating the ground truth of H&E stained slides having identified regions of tumor buds and/or identified regions that are not tumor buds comprises:
registering adjacent AE1/3 and H&E in order to provide a reliable ground truth for tumor regions and tumor budding;
utilizing swin transformers to segment a malignant tumor region on one or more H&E images; and
verifying the segmented malignant tumor regions found by the swin transformers on the one or more H&E images, wherein a series of automatic post-processing steps transfer small positive regions identified on AE1/3 (potential tumor buds) onto registered H&E, where they are filtered by number of nuclei to determine if they comprise a tumor bud.

### Claim 3 (depends on 1)

The method of claim 1, wherein the AI algorithm comprises a swin transformer.

### Claim 4 (depends on 1)

The method of claim 1, wherein the AI algorithm comprises a weakly-supervised machine learning method where labels are assigned to collections (called bags) rather than individual examples (called instances).

### Claim 5 (depends on 4)

The method of claim 4, wherein the weakly-supervised machine learning method comprises attention-based (AB) multiple-instance learning (MIL).

### Claim 6 (depends on 5)

The method of claim 5, wherein one or more bags are identified from the H&E stained slide image, each bag comprising tessellated whole-slide images (WSIs), in which each instance is a small unannotated image sampled from the WSI.

### Claim 7 (depends on 6)

The method of claim 6, wherein an attention-based pooling mechanism automatically learns to weight embedded instances into a bag-level feature vector dynamically, wherein a weight is automatically computed for each embedded instance, then a weighted sum combines them into a single, bag-level instance, corresponding to a slide-level embedding, and classification or regression is then performed on this bag-level embedding.

### Claim 8 (depends on 7)

The method of claim 7, wherein the attention-based pooling mechanism comprises a two-layer fully connected neural network that passes each instance embedding (hk) through two parallel layers of the network (V,U), applies a tanh and sigmoid activation function to the respective results, dots the results, then passes the fused activation through another layer (wT), which maps the vector into a single value, its attention weight (ak),
wherein
exp
⁢
{
w
T
⁢
tan
⁢
h
⁡
(
V
⁢
h
k
T
)
·
sig
⁢
m
⁡
(
U
⁢
h
k
T
)
}
∑
j
=
1
K
⁢
exp
⁢
{
w
T
⁢
tan
⁢
h
⁡
(
V
⁢
h
j
T
)
·
sig
⁢
m
⁡
(
U
⁢
h
k
T
)
}
.

### Claim 9 (depends on 8)

The method of claim 8, wherein a weighted sum of each embedded instance and its attention weight yields a bag-level instance (z), where
z
=
∑
k
=
1
K
⁢
a
k
⁢
h
k
.

### Claim 10 (depends on 8)

The method of claim 8, wherein parameters (V, U, w) for the two-layer neural network are automatically learned through model training.

### Claim 11 (depends on 10)

The method of claim 10, wherein the model training comprises using SqueezeNet pretrained on ImageNet as a feature extractor to generate instance embeddings (hk).

### Claim 12 (depends on 6)

The method of claim 6, wherein each bag comprises a 512×512 pixel section of the WSI, and each instance comprises a 96×96 pixel section of each bag.

### Claim 13 (depends on 4)

The method of claim 4, wherein a tumor budding label is assigned to each bag.

### Claim 14

A method for identifying tumor buds in an image of a hematoxylin & eosin (H&E) stained section of a tumor, the method comprising:
create a ground truth of H&E stained slides having identified regions of tumor buds and/or identified regions that are not tumor buds;
training an artificial intelligence (AI) algorithm using the dataset of H&E stained slides, wherein the AI algorithm comprises a weakly-supervised machine learning method where labels are assigned to bags comprising tessellated whole-slide images (WSIs) rather than instances comprising a small unannotated image sampled from the WSI;
analyze the image of the H&E stained section of the tumor using the trained AI algorithm to determine a presence or absence of tumor buds on the H&E stained slide.

### Claim 15 (depends on 14)

The method of claim 14, wherein the weakly-supervised machine learning method comprises attention-based (AB) multiple-instance learning (MIL).

### Claim 16 (depends on 15)

The method of claim 15, wherein an attention-based pooling mechanism automatically learns to weight embedded instances into a bag-level feature vector dynamically, wherein a weight is automatically computed for each embedded instance, then a weighted sum combines them into a single, bag-level instance, corresponding to a slide-level embedding, and classification or regression is then performed on this bag-level embedding.

### Claim 17 (depends on 16)

The method of claim 16, wherein the attention-based pooling mechanism comprises a two-layer fully connected neural network that passes each instance embedding (hk) through two parallel layers of the network (V,U), applies a tanh and sigmoid activation function to the respective results, dots the results, then passes the fused activation through another layer (wT), which maps the vector into a single value, its attention weight (ak), wherein
exp
⁢
{
w
T
⁢
tan
⁢
h
⁡
(
V
⁢
h
k
T
)
·
sigm
⁡
(
U
⁢
h
k
T
)
}
∑
j
=
1
K
⁢
exp
⁢
{
w
T
⁢
tan
⁢
h
⁡
(
V
⁢
h
j
T
)
·
sigm
⁡
(
U
⁢
h
k
T
)
}
.

### Claim 18 (depends on 17)

The method of claim 17, wherein a weighted sum of each embedded instance and its attention weight yields a bag-level instance (z), where
z
=
∑
k
=
1
K
⁢
a
k
⁢
h
k
.

### Claim 19 (depends on 17)

The method of claim 17, wherein parameters (V, U, w) for the two-layer neural network are automatically learned through model training.

### Claim 20 (depends on 19)

The method of claim 19, wherein the model training comprises using SqueezeNet pretrained on ImageNet as a feature extractor to generate instance embeddings (hk).

### Claim 21 (depends on 14)

The method of claim 14, wherein each bag comprises a 512×512 pixel section of the WSI, and each instance comprises a 96×96 pixel section of each bag.

### Claim 22 (depends on 14)

The method of claim 14, wherein a tumor budding label is assigned to each bag.
