# US 2023/0274534 A1 · Annotation refinement for segmentation of whole-slide images in digital pathology

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2023/0274534 A1
- Document ID: US20230274534A1
- Status: Published application
- Kind: A1
- Application number: 18/173,330
- Filed: 2023-02-23
- Published: 2023-08-31
- Earliest claimed: 2022-02-25
- Assignees: Siemens Healthcare GmbH
- Applicants: Siemens Healthcare GmbH
- Inventors: Andre AICHERT; Marvin TEICHMANN; Birgi TAMERSOY; Martin KRAUS; Arnaud Arindra ADIYOSO
- Web page: https://patentgrep.com/patent/US20230274534A1

## CPC

- G06V 10/774: Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
- G06F 18/00: Pattern recognition
- G06T 7/0012: {Biomedical image inspection}
- G06T 7/11: Region-based segmentation
- G06V 10/26: Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion
- G06V 10/56: relating to colour
- G06V 10/764: using classification, e.g. of video objects
- G06V 10/7715: {Feature extraction, e.g. by transforming the feature space, e.g. multi-dimensional scaling [MDS]; Mappings, e.g. subspace methods}
- G06V 20/695: {Preprocessing, e.g. image segmentation}
- G06V 20/70: Labelling scene content, e.g. deriving syntactic or semantic representations
- G06T 2207/20021: Dividing image into blocks, subimages or windows
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/30024: Cell structures in vitro; Tissue sections in vitro
- G06T 2207/30096: Tumor; Lesion
- G06V 2201/03: Recognition of patterns in medical or anatomical images
- G06V 2201/032: of protuberances, polyps nodules, etc.
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

Various disclosed examples pertain to digital pathology, more specifically to training of a segmentation algorithm for segmenting whole-slide images depicting tissue of multiple types. An initial annotation of a whole-slide image is refined to yield a refined annotation based on which parameters of the segmentation algorithm can be set. Techniques of patch-wise weak supervision can be employed for such refinement.

## Claims

### Claim 1

A computer-implemented method of training a segmentation algorithm for digital pathology, the segmentation algorithm configured to segment a whole-slide image depicting tissue of multiple types, in accordance with the multiple types, wherein the computer-implemented method comprises:
obtaining the whole-slide image;
obtaining a first annotation of at least a part of the whole-slide image and for the multiple types, the first annotation having a first level of detail;
determining, based on the first annotation, a second annotation of the whole-slide image and for the multiple types, the second annotation having a second level of detail which is higher than the first level of detail; and
setting parameters of the segmentation algorithm based on the second annotation.

### Claim 2 (depends on 1)

The computer-implemented method of claim 1, further comprising:
performing, based on the first annotation, a training of a classification algorithm configured to classify image regions of the whole-slide image in accordance with the multiple types; and
upon said performing of the training of the classification algorithm, using the classification algorithm to determine the second annotation.

### Claim 3 (depends on 2)

The computer-implemented method of claim 2, further comprising:
determining class activation maps of the classification algorithm for the image regions of the whole-slide image, and
wherein the second annotation is determined based on the class activation maps.

### Claim 4 (depends on 3)

The computer-implemented method of claim 3,
wherein the class activation maps have a third level of detail, which is smaller than the second level of detail.

### Claim 5 (depends on 3)

The computer-implemented method of claim 3, further comprising:
extracting, at the second level of detail, features from the whole-slide image, and
wherein the second annotation is determined based on the features and the class activation maps.

### Claim 6 (depends on 5)

The computer-implemented method of claim 5, further comprising:
determining, based on the features extracted from the whole-slide image, a partitioning of the whole-slide image,
wherein the second annotation is determined based on a combination of the partitioning and the class activation maps.

### Claim 7 (depends on 5)

The computer-implemented method of claim 5,
wherein the features include at least one of a color or a contrast gradient.

### Claim 8 (depends on 6)

The computer-implemented method of claim 6,
wherein the partitioning includes super-pixels of the whole-slide image.

### Claim 9 (depends on 6)

The computer-implemented method of claim 6,
wherein the second annotation is determined based on a pooling of values of the class activation maps for each type across partitions of the partitioning.

### Claim 10 (depends on 3)

The computer-implemented method of claim 3,
wherein the first annotation is restricted to a fraction of the whole-slide image,
wherein the image regions of the whole-slide image for which the class activation maps are determined are at least partially outside of the fraction of the whole-slide image.

### Claim 11 (depends on 2)

The computer-implemented method of claim 2,
wherein any given image region of the whole-slide image having a label of the first annotation that is associated with a first type of the multiple types does not include tissue fractions of other types of the multiple types larger than a size threshold, and
wherein a length scale of the image regions corresponds to the size threshold.

### Claim 12 (depends on 1)

The computer-implemented method of claim 1,
wherein the first annotation is restricted to a fraction of the whole-slide image, and
wherein the second annotation covers the whole-slide image.

### Claim 13 (depends on 1)

The computer-implemented method of claim 1,
wherein the first annotation is obtained from a manual annotation process.

### Claim 14 (depends on 1)

The computer-implemented method of claim 1,
wherein any given image region of the whole-slide image having a label of the first annotation that is associated with a first type of the multiple types does not include tissue fractions of other types of the multiple types beyond at least one of a quota or larger than a size threshold.

### Claim 15 (depends on 1)

The computer-implemented method of claim 1, further comprising:
obtaining a further whole-slide image;
determining a segmentation result using the segmentation algorithm for the further whole-slide image; and
detecting, based on the segmentation result, tumor-type tissue in the further whole-slide image.

### Claim 16 (depends on 4)

The computer-implemented method of claim 4, further comprising:
extracting, at the second level of detail, features from the whole-slide image,
wherein the second annotation is determined based on the features and the class activation map.

### Claim 17 (depends on 7)

The computer-implemented method of claim 7,
wherein the second annotation is determined based on a pooling of values of the class activation maps for each type across partitions of the partitioning.

### Claim 18 (depends on 8)

The computer-implemented method of claim 8,
wherein the second annotation is determined based on a pooling of values of the class activation maps for each type across partitions of the partitioning.

### Claim 19 (depends on 1)

A non-transitory computer-readable storage medium storing computer-executable instructions that, when executed by at least one processor, cause the at least one processor to perform the computer-implemented method of claim 1.

### Claim 20

A device for training a segmentation algorithm for digital pathology, the segmentation algorithm configured to segment a whole-slide image depicting tissue of multiple types, in accordance with the multiple types, wherein the device comprises:
at least one processor; and
at least one memory storing computer-executable instructions that, when executed at the least one processor, cause the device to obtain the whole-slide image,
obtain a first annotation of at least a part of the whole-slide image and for the multiple types, the first annotation having a first level of detail,
determine, based on the first annotation, a second annotation of the whole-slide image and for the multiple types, the second annotation having a second level of detail which is higher than the first level of detail, and
set parameters of the segmentation algorithm based on the second annotation.
