# US 12,118,455 B2 · Systems, methods, and/or media, for selecting candidates for annotation for use in training a classifier

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,118,455 B2
- Document ID: US12118455B2
- Status: Granted
- Kind: B2
- Application number: 15/965,691
- Filed: 2018-04-27
- Published: 2018-11-01
- Granted: 2024-10-15
- Earliest claimed: 2017-04-27
- Assignees: Arizona Board of Regents on behalf of Arizona State University
- Applicants: Arizona Board of Regents On Behalf of Arizona State University
- Inventors: Jianming Liang; Zongwei Zhou; Jae Shin
- Web page: https://patentgrep.com/patent/US12118455B2

## CPC

- G06N 3/08: Learning methods
- G06F 18/2148: {characterised by the process organisation or structure, e.g. boosting cascade}
- G06F 18/217: {Validation; Performance evaluation; Active pattern learning techniques}
- G06F 18/2413: based on distances to training or reference patterns
- G06F 18/28: Determining representative reference patterns, e.g. by averaging or distorting; Generating dictionaries
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/09: Supervised learning
- G06N 3/091: Active learning
- G06N 3/096: Transfer learning
- G06V 10/454: {Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]}
- G06V 10/764: using classification, e.g. of video objects
- G06V 10/772: Determining representative reference patterns, e.g. averaging or distorting patterns; Generating dictionaries
- G06V 10/7747: {Organisation of the process, e.g. bagging or boosting}
- G06V 10/776: Validation; Performance evaluation
- G06V 10/82: using neural networks
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

Systems for selecting candidates for labelling and use in training a convolutional neural network (CNN) are provided, the systems comprising: a memory device; and at least one hardware processor configured to: receive a plurality of input candidates, wherein each candidate includes a plurality of identically labelled patches; and for each of the plurality of candidates: determine a plurality of probabilities, each of the plurality of probabilities being a probability that a unique patch of the plurality of identically labelled patches of the candidate corresponds to a label using a pre-trained CNN; identify a subset of candidates of the plurality of input candidates, wherein the subset does not include all of the plurality of candidates, based on the determined probabilities; query an external source to label the subset of candidates to produce labelled candidates; and train the pre-trained CNN using the labelled candidates.

## Claims

### Claim 1

A system for selecting and labeling a Region of Interest (ROI) in an original medical image by fine-tuning a pre-trained convolutional neural network (CNN) using annotated medical images, wherein the system comprises:
a memory to store instructions;
a set of one or more hardware processors configured to execute the instructions to cause the system to perform operations including:
receiving the annotated medical images for use in fine-tuning the pre-trained CNN;
generating a plurality of input candidates for fine-tuning the pre-trained CNN from the annotated medical images by performing one or more data augmentation operations, each of the plurality of input candidates for use in fine-tuning the pre-trained CNN having multiple identically labeled patches derived from the annotated medical images;
providing the plurality of input candidates having the plurality of identically labeled patches as input to the pre-trained CNN;
executing the pre-trained CNN to determine a plurality of probabilities for each of the plurality of input candidates, wherein each of the plurality of probabilities define a likelihood that a unique patch amongst the plurality of identically labeled patches of the respective input candidate corresponds to a label associated with annotations for the annotated medical images from which the respective input candidate was derived;
determining an entropy value for each of the plurality of input candidates based on the plurality of probabilities of the corresponding input candidate;
sorting the plurality of input candidates according to their respective entropy value;
identifying a subset of top-ranked input candidates amongst the sorted plurality of input candidates, wherein the subset does not include all of the plurality of input candidates;
labeling the subset of top-ranked input candidates to produce labeled candidates;
fine-tuning the pre-trained CNN using the labeled candidates to produce a trained CNN; and
selecting and labeling the ROI within the original medical image and identifying via the trained CNN a location for the ROI within the original medical image using an ROI object confidence map produced by the trained CNN, wherein the location of the ROI is determined by identifying a largest connected object in the original medical image which is connected to an identified object within the ROI object confidence map based on at least one pixel of an object in the original medical image falling within the ROI; and
returning as output, the selected and labeled ROI for the original medical image.

### Claim 2 (depends on 1)

The system of claim 1, wherein the set of one or more hardware processors is further configured to execute the instructions to cause the system to perform operations including:
determining an average of the plurality of probabilities for each of the plurality of candidates; and
select a top percentage of the plurality of identically labeled patches for a candidate when the average is greater than a threshold.

### Claim 3 (depends on 1)

The system of claim 1, wherein the one or more data augmentation operations comprise:
(i) cropping the annotated medical image into multiple parts, each cropped part having a label based on the annotated medical image, (ii) translating each cropped part into a group of multiple identically labeled patches sharing a same label, and (iii) naming each respective group of multiple identically labeled patches as one of the plurality of input candidates for use in training the CNN.

### Claim 4 (depends on 1)

The system of claim 1:
wherein identifying the largest connected object in the original medical image which is connected to the identified object within the ROI object confidence map comprises executing an indicator function;
wherein the indicator function excludes all pixels located farther than a threshold distance from a location of the object in the original medical image; and
wherein the at least one pixel of the object within the original medical image falling within the ROI comprises the at least one pixel falling within the threshold distance from the location of the object in the original medical image.

### Claim 5 (depends on 1)

The system of claim 1, wherein
entropy for each of the plurality of input candidates based on the plurality of probabilities of the corresponding input candidate
comprises determining the entropy εi of the corresponding candidate i using the following equation:
ε
i
=
-
1
m
⁢
∑
j
=
1
m
∑
k
=
1
❘
"\[LeftBracketingBar]"
Y
❘
"\[RightBracketingBar]"
p
i
j
,
k
⁢
log
⁢
p
i
j
,
k
where:
m is the number of patches for the corresponding candidate;
|Y| is the number of possible labels; and
pij,k is the probability that patch j candidate i corresponds to label k.

### Claim 6 (depends on 1)

The system of claim 1:
wherein the original medical image forms no part of the annotated medical images used for fine-tuning the pre-trained CNN;
wherein the ROI object confidence map comprises an ROI and bulb confidence map identifying the object in the original medical image; and
wherein the object in the original medical image comprises a carotid bulb selected and labeled within the ROI.

### Claim 7 (depends on 1)

The system of claim 1, wherein the set of one or more hardware processors is also configured to execute the instructions to cause the system to perform the operations including:
determining a diversity value d, for each of the plurality of input candidates based on the plurality of probabilities of the corresponding input candidate;
wherein sorting the plurality of input candidates according to their respective entropy value comprises sorting the plurality of input candidates further according to their respective diversity value; and
wherein the diversity value di for a corresponding input candidate i is calculated using the following equation:
d
i
=
∑
k
=
1
❘
"\[LeftBracketingBar]"
Y
❘
"\[RightBracketingBar]"
∑
j
=
1
m
∑
l
=
j
m
(
p
i
j
,
k
-
p
i
l
,
k
)
⁢
log
⁢
p
i
j
,
k
p
i
l
,
k
;
where:
m is the number of patches for the corresponding candidate;
|Y| is the number of possible labels;
pij,k is the probability that patch j candidate i corresponds to label k; and
pil,k is the probability that patch l candidate i corresponds to label k.

### Claim 8 (depends on 1)

The system of claim 1, wherein the subset of top-ranked input candidates are misclassified candidates.

### Claim 9 (depends on 1)

The system of claim 1, wherein the set of one or more hardware processors is also configured to form a set of labeled candidates from previously labeled candidates and the labeled candidates produced in response to a query.

### Claim 10 (depends on 1)

The system of claim 1:
wherein each one of the plurality of input candidates constitutes a single Annotation Unit (AU); and
wherein the method further comprises labeling the entirety of each single AU as either informative or non-informative.

### Claim 11 (depends on 1)

The system of claim 1, wherein translating each cropped part into the group of multiple identically labeled patches comprises at least one of:
(i) enlarging each cropped part into a larger patch when compared with an original size for the cropped part and re-cropping the larger patch back to the original size to yield a new patch as part of performing the data augmentation operation;
(ii) increasing a pixel size of each cropped part to form an increased pixel size patch when compared with an original size for the cropped part and re-cropping the increased pixel size patch to a centered pre-defined quantity of pixels as a new patch after performing the data augmentation operation;
(iii) applying a configurable percentage of a resized bounding box to each cropped part in vertical and horizontal directions to yield a new patch as part of performing the data augmentation operation;
(iv) applying a rotation based data augmentation operation at an identified center of a polyp location within one of the cropped parts to yield a new patch as part of performing the data augmentation operation;
(v) applying a rotation-plus-scale based data augmentation operation by extracting multiple different physical sizes from one of the cropped parts and by further rotating longitudinal and cross-sectional vessel planes around a vessel axis to yield a new patch as part of performing the data augmentation operation; and
(vi) rotating one of the cropped parts eight times by mirroring and flipping the respective cropped part into mirrored and flipped translations to yield new patches as part of performing the data augmentation operation.

### Claim 12

A method for selecting and labeling a Region of Interest (ROI) in an original medical image by fine-tuning a pre-trained convolutional neural network (CNN) using annotated medical images, wherein the method comprises:
receiving the annotated medical images for use in fine-tuning the pre-trained CNN;
generating a plurality of input candidates for fine-tuning the pre-trained CNN from the annotated medical images by performing one or more data augmentation operations, each of the plurality of input candidates for use in fine-tuning the pre-trained CNN having multiple identically labeled patches derived from the annotated medical images;
providing the plurality of input candidates having the plurality of identically labeled patches as input to the pre-trained CNN;
executing the pre-trained CNN to determine a plurality of probabilities for each of the plurality of input candidates, wherein each of the plurality of probabilities define a likelihood that a unique patch amongst the plurality of identically labeled patches of the respective input candidate corresponds to a label associated with annotations for the annotated medical images from which the respective input candidate was derived;
determining an entropy value for each of the plurality of input candidates based of the plurality of probabilities of the corresponding input candidate;
sorting the plurality of input candidates ding to their respective entropy value;
identifying a subset of top-ranked input candidates amongst the sorted plurality of input candidates, wherein the subset does not include all of the plurality of input candidates;
labeling the subset of top-ranked input candidates to produce labeled candidates;
fine-tuning the pre-trained CNN using the labeled candidates to produce a trained CNN; and
selecting and labeling the ROI within the original medical image and identifying via the trained CNN a location for the ROI within the original medical image using an ROI object confidence map produced by the trained CNN, wherein the location of the ROI is determined by identifying a largest connected object in the original medical image which is connected to an identified object within the ROI object confidence map based on at least one pixel of an object in the original medical image falling within the ROI; and
returning as output, the selected and labeled ROI for the original medical image.

### Claim 13 (depends on 12)

The method of claim 12, further comprising:
determining an average of the plurality of probabilities for each of the plurality of candidates; and
selecting a top percentage of the plurality of identically labeled patches for a candidate when the average is greater than a threshold.

### Claim 14 (depends on 12)

The method of claim 12,
wherein the entropy εi of for the corresponding input candidate i is determined using the following equation:
ε
i
=
-
1
m
⁢
∑
j
=
1
m
∑
k
=
1
❘
"\[LeftBracketingBar]"
Y
❘
"\[RightBracketingBar]"
p
i
j
,
k
⁢
log
⁢
p
i
j
,
k
where:
m is the number of patches for the corresponding candidate;
|Y| is the number of possible labels; and
pij,k is the probability that patch j candidate i corresponds to label k.

### Claim 15 (depends on 12)

The method of claim 12:
wherein identifying the largest connected object in the original medical image which is connected to the identified object within the ROI object confidence map comprises executing an indicator function;
wherein the indicator function excludes all pixels located farther than a threshold distance from a location of the object in the original medical image; and
wherein the at least one pixel of the object within the original medical image falling within the ROI comprises the at least one pixel falling within the threshold distance from the location of the object in the original medical image.

### Claim 16 (depends on 12)

The method of claim 12:
wherein the original medical image forms no part of the annotated medical images used for fine-tuning the pre-trained CNN;
wherein the ROI object confidence map comprises an ROI and bulb confidence map identifying the object within in the original medical image; and
wherein the object in the original medical image comprises a carotid bulb selected and labeled within the ROI.

### Claim 17 (depends on 12)

The method of claim 12, further comprising:
determining a diversity value di for each of the plurality of input candidates based on the plurality of probabilities of the corresponding input candidate;
wherein sorting the plurality of input candidates cording to their respective entropy value comprises sorting the plurality of input candidates further according to their respective diversity value; and
wherein the diversity value di for a corresponding input candidate i is determined using the following equation:
d
i
=
∑
k
=
1
❘
"\[LeftBracketingBar]"
Y
❘
"\[RightBracketingBar]"
∑
j
=
1
m
∑
l
=
j
m
(
p
i
j
,
k
-
p
i
l
,
k
)
⁢
log
⁢
p
i
j
,
k
p
i
l
,
k
;
where:
m is the number of patches for the corresponding candidate;
|Y| is the number of possible labels;
pij,k is the probability that patch j candidate i corresponds to label k; and
pil,k is the probability that patch l candidate i corresponds to label k.

### Claim 18 (depends on 12)

The method of claim 12, wherein the subset of top-ranked input candidates are misclassified candidates.

### Claim 19 (depends on 12)

The method of claim 12, further comprising forming a set of labeled candidates from previously labeled candidates and the labeled candidates produced in response to a query.

### Claim 20

Non-transitory computer readable storage media having instructions stored thereupon that, when executed by a processor and a memory of a system, the instructions cause the system to perform operations comprising:
receiving annotated medical images for use in fine-tuning a pre-trained CNN;
generating a plurality of input candidates for fine-tuning the pre-trained CNN from the annotated medical images by performing one or more data augmentation operations, each of the plurality of input candidates for use in fine-tuning the pre-trained CNN having multiple identically labeled patches derived from the annotated medical images;
providing the plurality of input candidates having the plurality of identically labeled patches as input to the pre-trained CNN;
executing the pre-trained CNN to determine a plurality of probabilities for each of the plurality of input candidates, wherein each of the plurality of probabilities define a likelihood that a unique patch amongst the plurality of identically labeled patches of the respective input candidate corresponds to a label associated with annotations for the annotated medical images from which the respective input candidate was derived;
determining an entropy value for each of the plurality of input candidates based on the plurality of probabilities of the corresponding input candidate;
sorting the plurality of input candidates according to their respective entropy value;
identifying a subset of top-ranked input candidates amongst the sorted plurality of input candidates, wherein the subset does not include all of the plurality of input candidates;
labeling the subset of top-ranked input candidates to produce labeled candidates;
fine-tuning the pre-trained CNN using the labeled candidates to produce a trained CNN; and
selecting and labeling a region of interest (ROI) within the annotated medical images and identifying via the trained CNN a location for the ROI within the annotated medical images using an ROI object confidence map produced by the trained CNN, wherein the location of the ROI is determined by identifying a largest connected object in the annotated medical images which is connected to an identified object within the ROI object confidence map based on at least one pixel of an object in the annotated medical images falling within the ROI; and
returning as output, the selected and labeled ROI for the annotated medical images.