# US 12,008,748 B2 · Method for classifying fundus image of subject and device using same

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,008,748 B2
- Document ID: US12008748B2
- Status: Granted
- Kind: B2
- Application number: 17/059,868
- Filed: 2019-04-10
- Published: 2021-07-22
- Granted: 2024-06-11
- Earliest claimed: 2018-05-31
- Assignees: VUNO Inc.
- Applicants: Vuno, Inc.
- Inventors: Sang Keun Kim; Hyun-Jun Kim; Kyuhwan Jung; Jae Min Son
- Web page: https://patentgrep.com/patent/US12008748B2

## CPC

- G06T 7/0012: {Biomedical image inspection}
- G16H 20/40: relating to mechanical, radiation or invasive therapies, e.g. surgery, laser therapy, dialysis or acupuncture
- G06T 7/187: involving region growing; involving region merging; involving connected component labelling
- G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80)
- G16H 30/20: for handling medical images, e.g. DICOM, HL7 or PACS
- A61B 3/12: for looking at the eye fundus, e.g. ophthalmoscopes (A61B3/13 takes precedence)
- A61B 3/14: Arrangements specially adapted for eye photography
- G06T 2207/10024: Color image
- G06T 2207/10081: Computed x-ray tomography [CT]
- G06T 2207/10088: Magnetic resonance imaging [MRI]
- G06T 2207/10104: Positron emission tomography [PET]
- G06T 2207/10108: Single photon emission computed tomography [SPECT]
- G06T 2207/10116: X-ray image
- G06T 2207/10136: 3D ultrasound image
- G06T 2207/20076: Probabilistic image processing
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/20132: Image cropping
- G06T 2207/30041: Eye; Retina; Ophthalmic
- G06T 2207/30096: Tumor; Lesion
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G16H: HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
- A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D)

## Abstract

The present invention relates to a method for classifying a fundus image and a device using same. Specifically, according to the method of the present invention, a computing device acquires a fundus image of a subject, generates classification information of the fundus image, generates an interpretation text on the basis of the classification information, and provides the interpretation text to an external entity.

## Claims

### Claim 1

A method of classifying an image using a deep neural network, the method comprising:
acquiring, by a computing apparatus, the image;
generating, by a computing apparatus, classification information of the image based on the deep neural network; and
providing, by a computing apparatus, the classification information to an external entity,
wherein the classification information comprises a classification to which the image belongs and an activation map generated based on information of individual findings in the image related to the classification, and
wherein the activation map includes visualization information indicating at least one location related to the individual findings in the image.

### Claim 2 (depends on 1)

The method of claim 1, wherein the classification to which the image belongs is determined as at least one of a classification in which the image is non-interpretable, a classification in which the image is normal, and a classification in which the image is abnormal.

### Claim 3 (depends on 2)

The method of claim 2, wherein, when the image belongs to the classification in which the image is abnormal, the activation map is generated based on locations of the individual findings.

### Claim 4 (depends on 2)

The method of claim 2, wherein, when the image belongs to the classification in which the image is non-interpretable, an indication corresponding to “ungradable” is provided to the external entity.

### Claim 5 (depends on 2)

The method of claim 2, wherein the classification information further comprises confidential level information regarding the classification in which the image is normal or the classification in which the image is abnormal.

### Claim 6 (depends on 1)

The method of claim 1, further comprising:
providing, by the computing apparatus, an interface of modifying and editing the classification information.

### Claim 7 (depends on 1)

The method of claim 1, wherein the deep neural network is pretrained to decrease a value of a loss function calculated based on an input learning image, and
the loss function is calculated based on a combination of a classification loss function that is calculated based on classification results of the deep neural network and a guidance loss function that is calculated based on the activation map.

### Claim 8 (depends on 7)

The method of claim 7, wherein the guidance loss function is calculated based on an operation using a value of a first pixel comprised in a region mask provided as a label corresponding to the activation map and a value of a second pixel comprised in the activation map, and
a location of the first pixel is determined as a location on the region mask corresponding to a location of the second pixel in the activation map.

### Claim 9 (depends on 8)

The method of claim 8, wherein the guidance loss function is calculated based on a multiplication of a first function that is calculated based on the value of the first pixel and a second function that is calculated based on the value of the second pixel.

### Claim 10 (depends on 7)

The method of claim 7, wherein the guidance loss function is represented as
L
guide
(
A
,
M
)
=
1
k
⁢
W
F
⁢
H
F
⁢
∑
i
=
1
k
∑
l
=
1
W
F
⁢
H
F
(
1
-
m
l
i
)
⁢
f
⁡
(
a
l
i
+
ϵ
)
where the k denotes a number of pieces of mini-batch data that at least a portion of learning data is extracted as mini-batch learning data, the WF is a natural number and denotes a width of an activation map about target findings, the HF is a natural number and denotes a height of the activation map, the A={A1, . . . , Ak} denotes the activation map about the target findings of the k mini-batch data where each of the A1, . . . , Ak denotes a single element selected from a Descartes product of WF×HF dimension of a set (0, 1) having real numbers between 0 and 1 as elements, the M={M1, . . . , Mk} denotes a region mask about the target findings of the k mini-batch data where each of the M1, . . . , Mk denotes a single element selected from a Descartes product of WF×HF dimension of a set {0, 1} having 0 and 1 as elements, the mli and the ali, respectively, denote values of lth pixels of the Mi and the Ai for each natural number 1 greater than or equal to 1 and less than or equal to WF×HF, the Ai denotes a positive number introduced for a numerical stability, and the f denotes a function that is determined based on a log function.

### Claim 11 (depends on 10)

The method of claim 10, wherein the f(ali,∈) corresponds to one of log (ali+∈) and log(max(ali+∈)).

### Claim 12 (depends on 1)

The method of claim 1, wherein the deep neural network comprises a plurality of convolution layers, and comprises an output layer comprising a first output layer for outputting a guidance loss and a second output layer for outputting a classification loss as at least two output layers that follow a final 1×1 convolution layer.

### Claim 13 (depends on 12)

The method of claim 12, wherein the first output layer calculates the classification through application of a sigmoid function after global average pooling (GAP) of the final 1×1 convolution layer, and
the second output layer calculates the activation map through the sigmoid function application of the final 1×1 convolution layer.

### Claim 14 (depends on 1)

The method of claim 1, wherein, before the image is input to the deep neural network, the computing apparatus performs preprocessing of cropping and resizing the image such that a fundus portion is positioned at a center of the image and the image is resized to have a size of between 256×256 and 1024×1024 and normalizing a pixel value of the image such that the pixel value of the image belongs to the range of 0 or more and 1 or less.

### Claim 15 (depends on 14)

The method of claim 14, wherein, if the size of the image exceeds 512×512, a resolution of the image decreases for a size of 256×256 through a convolution of consecutive stride 2.

### Claim 16 (depends on 1)

A non-transitory computer-readable record medium storing instructions that, when executed by a computing apparatus, cause the computing apparatus to perform the method of claim 1.

### Claim 17

A computing apparatus for classifying an image using a deep neural network, the computing apparatus comprising:
a communicator configured to acquire the image; and
a processor configured to perform a process of generating classification information of the image based on the deep neural network and a process of providing the classification information to an external entity,
wherein the classification information comprises a classification to which the image belongs and an activation map generated based on information of individual findings in the image related to the classification, and
wherein the activation map includes visualization information indicating at least one location related to the individual findings in the image.