# US 2022/0019860 A1 · Interactive iterative image annotation

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2022/0019860 A1
- Document ID: US20220019860A1
- Status: Published application
- Kind: A1
- Application number: 17/294,731
- Filed: 2019-11-15
- Published: 2022-01-20
- Earliest claimed: 2018-11-22
- Assignees: Koninklijke Philips N.V.
- Applicants: Koninklijke Philips N.V.
- Inventors: JUERGEN WEESE; THOMAS BLAFFERT; TOM BROSCH; HANS BARSCHDORF
- Web page: https://patentgrep.com/patent/US20220019860A1

## CPC

- G06F 18/2178: {based on feedback of a supervisor}
- G16H 50/20: for computer-aided diagnosis, e.g. based on medical expert systems
- G06F 18/2148: {characterised by the process organisation or structure, e.g. boosting cascade}
- G06F 18/40: Software arrangements specially adapted for pattern recognition, e.g. user interfaces or toolboxes therefor
- G06N 20/00: Machine learning
- G06V 10/454: {Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]}
- G06V 10/82: using neural networks
- G06T 2207/20081: Training; Learning
- G06V 2201/03: Recognition of patterns in medical or anatomical images
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G16H: HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

A system and computer-implemented method are provided for annotation of image data. A user is enabled to iteratively annotate the image data. An iteration of said iterative annotation comprises generating labels for a current image data part based on user-verified labels of a previous image data part, and enabling the user to verify and correct said generated labels to obtain user-verified labels for the current image data part. The labels for the current image data part are generated by combining respective outputs of a label propagation algorithm and a machine-learned classifier trained on user-verified labels and image data and applied to image data of the current image data part. The machine-learned classifier is retrained using the user-verified labels and the image data of the current image data part to obtain a retrained machine-learned classifier.

## Claims

### Claim 1

A system for annotation of image data, the system comprising:
an input interface configured to access the image data to be annotated;
a user interface subsystem comprising:
a user input interface configured to receive user input data from a user input device operable by a user;
a display output interface configured to provide display data to a display to visualize output of the system;
a processor configured to, using the user interface subsystem, establish a user interface which enables the user to iteratively annotate the image data, wherein an iteration of said iterative annotation comprises:
the processor generating labels for a current image data part based on user-verified labels of a previous image data part;
via the user interface, enabling the user to verify and correct said generated labels to obtain user-verified labels for the current image data part;
wherein the processor is further configured to:
generate the labels for the current image data part by combining, by weighting, respective outputs of:
a label propagation algorithm which propagates the user-verified labels of the previous image data part to the current image data part, and
a machine-learned classifier for labeling of image data, wherein the machine-learned classifier is trained on user-verified labels and image data and applied to image data of the current image data part; and
retrain the machine-learned classifier using the user-verified labels and the image data of the current image data part to obtain a retrained machine-learned classifier.

### Claim 2 (depends on 1)

The system according to claim 1, wherein the processor is configured to retrain the machine-learned classifier between iterations of the iterative annotation, or between the iterative annotation of different image data.

### Claim 3 (depends on 1)

The system according to claim 1, wherein the processor is configured to adjust the weighting during the iterative annotation of the image data, or between the iterative annotation of different image data.

### Claim 4 (depends on 1)

The system according to claim 1, wherein the processor is configured to determine the weighting based on a metric quantifying an annotation accuracy of the label propagation algorithm and/or the machine-learned classifier.

### Claim 5 (depends on 4)

The system according to claim 4, wherein the metric quantifies the annotation accuracy based on a difference between i) the output of the label propagation algorithm and/or the output of the machine-learned classifier and ii) the user-corrected labels.

### Claim 6 (depends on 3)

The system (100) according to claim 3, wherein the processor is configured to adjust the weighting by increasing a weighting of the output of the machine-learned classifier relative to the output of the label propagation algorithm.

### Claim 7 (depends on 6)

The system according to claim 6, wherein the processor is configured to start the weighting of the output of the machine-learned classifier at or substantially at zero at a start of the iterative annotation.

### Claim 8 (depends on 1)

The system according to claim 1, wherein the weighting comprises a global weighting per image data part and/or a local weighting per pixel, voxel or other image sub-region.

### Claim 9 (depends on 1)

The system according to claim 1, wherein the output of the label propagation algorithm and/or the output of the machine-learned classifier is a probability map, or one or more control points defining a contour.

### Claim 10 (depends on 1)

The system according to claim 1, wherein the label propagation algorithm is configured to propagate the user-verified labels of the previous image data part to the current image data part based on a similarity in image data between the previous image data part and the current image data part.

### Claim 11 (depends on 10)

The system according to claim 10, wherein the label propagation algorithm is a patch-based label propagation algorithm.

### Claim 12 (depends on 1)

A workstation or imaging apparatus comprising the system according to claim 1.

### Claim 13

A computer-implemented method for annotation of image data, the method comprising:
accessing the image data to be annotated;
using a user interface, enabling a user to iteratively annotate the image data, wherein an iteration of said iterative annotation comprises:
generating labels for a current image data part based on user-verified labels of a previous image data part;
via the user interface, enabling the user to verify and correct said generated labels to obtain user-verified labels for the current image data part;
wherein the labels for the current image data part are generated by combining, by weighting, respective outputs of:
a label propagation algorithm which propagates the user-verified labels of the previous image data part to the current image data part, and
a machine-learned classifier for labeling of image data, wherein the machine-learned classifier is trained on user-verified labels and image data and applied to image data of the current image data part; and
retraining the machine-learned classifier using the user-verified labels and the image data of the current image data part to obtain a retrained machine-learned classifier.

### Claim 14

(canceled)

### Claim 15

A non-transitory computer readable medium storing instructions that, when executed by one or more processors, cause the one or more processors to:
access image data to be annotated;
using a user interface, enable a user to iteratively annotate the image data, wherein an iteration of said iterative annotation comprises:
generating labels for a current image data part based on user-verified labels of a previous image data part;
via the user interface, enabling the user to verify and correct said generated labels to obtain user-verified labels for the current image data part;
wherein the labels for the current image data part are generated by combining, by weighting, respective outputs of:
a label propagation algorithm which propagates the user-verified labels of the previous image data part to the current image data part, and
a machine-learned classifier for labeling of image data, wherein the machine-learned classifier is trained on user-verified labels and image data and applied to image data of the current image data part; and
retraining the machine-learned classifier using the user-verified labels and the image data of the current image data part to obtain a retrained machine-learned classifier.

### Claim 16 (depends on 15)

The non-transitory computer readable medium of claim 15, wherein the machine-learned classifier is retrained between iterations of the iterative annotation, or between the iterative annotation of different image data.

### Claim 17 (depends on 15)

The non-transitory computer readable medium of claim 15, wherein weighting is adjusted during the iterative annotation of the image data or between the iterative annotation of different image data.

### Claim 18 (depends on 15)

The non-transitory computer readable medium of claim 15, wherein the weighting is determined based on a metric quantifying an annotation accuracy of the label propagation algorithm and/or the machine-learned classifier.

### Claim 19 (depends on 18)

The non-transitory computer readable medium of claim 18, wherein the metric quantifies the annotation accuracy based on a difference between i) the output of the label propagation algorithm and/or the output of the machine-learned classifier and ii) the user-corrected labels.

### Claim 20 (depends on 17)

The non-transitory computer readable medium of claim 17, wherein the weighting is adjusted by increasing a weighting of the output of the machine-learned classifier relative to the output of the label propagation algorithm.

### Claim 21 (depends on 20)

The non-transitory computer readable medium of claim 20, wherein the weighting of the output of the machine-learned classifier is started at or substantially at zero at a start of the iterative annotation.
