# US 2019/0171915 A1 · Systems and user interfaces for enhancement of data utilized in machine-learning based medical image review

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2019/0171915 A1
- Document ID: US20190171915A1
- Status: Published application
- Kind: A1
- Application number: 15/831,061
- Filed: 2017-12-04
- Published: 2019-06-06
- Earliest claimed: 2017-12-04
- Assignees: International Business Machines Corporation
- Applicants: International Business Machines Corporation
- Inventors: Murray A. Reicher; Aviad Zlotnick
- Web page: https://patentgrep.com/patent/US20190171915A1

## CPC

- G06V 10/7788: {the supervisor being a human, e.g. interactive learning with a human teacher}
- G06F 3/04842: Selection of displayed objects or displayed text elements (G06F3/0482 takes precedence)
- G06F 18/217: {Validation; Performance evaluation; Active pattern learning techniques}
- G06F 18/24: Classification techniques
- G06F 18/41: {Interactive pattern learning with a human teacher}
- G06V 30/194: References adjustable by an adaptive method, e.g. learning
- G06V 2201/03: Recognition of patterns in medical or anatomical images
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

Systems and techniques are disclosed for improvement of machine learning systems based on enhanced training data. An example method includes providing a visual concurrent display of a set of images of features, the features requiring classification by a reviewing user. The user interface is provided to enable the reviewing user to assign classifications to the images, the user interface being configured to create, read, update, and/or delete classifications. The user interface is responsive to the user, with the user response indicating at least two images with a single classification. The user interface is updated to represent the single classification.

## Claims

### Claim 1

A method for facilitating proper classification of features in medical images comprising:
generating, for display on a display of a computing system, an interactive classification user interface concurrently displaying each of a plurality of medical images each including an object to be classified by a user of the computing system, at least some of the medical images representing only portions of respective medical images including objects to be classified;
the interactive classification user interface enabling the user to select subsets of the images to be concurrently assigned classifications, each classification indicating an attribute of the image, such as shape, size, malignancy risk, or clinical relevance;
providing the classifications of the objects to a machine learning system.

### Claim 2 (depends on 1)

The method of claim 1, wherein borders of the objects included in the medical images are highlighted.

### Claim 3 (depends on 1)

The method of claim 1, wherein the interactive classification user interface displays the medical images as a grid.

### Claim 4 (depends on 1)

The method of claim 1, wherein the interactive classification user interface displays the medical images as a stack.

### Claim 5 (depends on 4)

The method of claim 4, wherein the medical images displayed as a stack cycle periodically, such that a top presented medical image on the stack is replaced with a subsequent medical image after a threshold amount of time.

### Claim 6 (depends on 1)

The method of claim 1, wherein the classifications assigned by the user are stored, and wherein the classifications are associated with the user.

### Claim 7 (depends on 1)

The method of claim 1, wherein a threshold number of users assign classifications to the medical images, and wherein a final classification is determined for each medical image prior to providing the classifications to the machine learning system.

### Claim 8 (depends on 7)

The method of claim 7, wherein the determination of a final classification is based at least in part on a performance associated with each user, the performance being based on a performance in correctly classifying control medical images of known classifications.

### Claim 9 (depends on 1)

The method of claim 1, wherein the machine learning system is configured to update an object detection model based on the user classifications, the object detection model being usable to detect classify other medical images as including objects with particular shape, size, malignancy risk, clinical relevance, or other features based on similarities to objects classified by the user.

### Claim 10

Non-transitory computer storage-media storing instructions that when executed by a system of one or more computers, cause the one or more computers to perform operations comprising:
providing a visual concurrent display of a set of images of features, the features requiring classification by a reviewing user;
providing a user interface enabling the reviewing user to assign classifications to the images, the user interface being configured to create, read, update, and/or delete classifications; and
responsive to the user utilizing the user interface to indicate at least two images with a single classification, updating the at least two images to represent the single classification.

### Claim 11 (depends on 10)

The computer storage-media of claim 10, wherein the operations further comprise:
training a machine learning system based on marking the at least two images with the single classification.

### Claim 12 (depends on 10)

The computer storage-media of claim 10, wherein the visual concurrent display is a grid

### Claim 13 (depends on 10)

The computer storage-media of claim 10, wherein the visual concurrent display is stacked.

### Claim 14 (depends on 10)

The computer storage-media of claim 10, wherein the images are medical images and the classification is selected from a group consisting of a finding type, shape, border type, homogeneity, BIRADS score.

### Claim 15 (depends on 10)

The computer storage-media of claim 10, wherein the classifications assigned by the reviewing user are stored, and wherein the classifications are associated with the reviewing user.

### Claim 16 (depends on 10)

The computer storage-media of claim 10, wherein a threshold number of reviewing users assign classifications to the images, and wherein a final classification is determined for each image.

### Claim 17

A system of one or more computers and non-transitory computer storage media storing instructions that when executed by the one or more computers, cause the one or more computers to perform operations comprising:
providing a visual concurrent display of a set of images of features, the features requiring classification by a reviewing user;
providing a user interface enabling the reviewing user to assign classifications to the images, the user interface being configured to create, read, update, and/or delete classifications; and
responsive to the user utilizing the user interface to indicate at least two images with a single classification, updating the at least two images to represent the single classification.

### Claim 18 (depends on 17)

The system of claim 17, wherein the operations further comprise:
training a machine learning system based on marking the at least two images with the single classification.

### Claim 19 (depends on 17)

The system of claim 17, wherein the visual concurrent display is a grid or a stack.

### Claim 20 (depends on 17)

The system of claim 17, wherein the images are medical images and the classification is selected from a group consisting of a finding type, shape, border type, homogeneity, BIRADS score.
