# US 12,009,105 B2 · Learning apparatus and learning method for training neural network

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,009,105 B2
- Document ID: US12009105B2
- Status: Granted
- Kind: B2
- Application number: 17/197,060
- Filed: 2021-03-10
- Published: 2021-07-15
- Granted: 2024-06-11
- Earliest claimed: 2018-09-20
- Assignees: FUJIFILM Corporation
- Applicants: FUJIFILM Corporation
- Inventors: Masaaki Oosake
- Web page: https://patentgrep.com/patent/US12009105B2

## CPC

- G16H 50/70: for mining of medical data, e.g. analysing previous cases of other patients
- A61B 1/000094: {extracting biological structures}
- A61B 1/000096: {using artificial intelligence}
- A61B 6/00: Apparatus or devices for radiation diagnosis; Apparatus or devices for radiation diagnosis combined with radiation therapy equipment (instruments measuring radiation intensity for application in the field of nuclear medicine, e.g. in vivo counting, G01T1/161)
- A61B 6/03: Computed tomography [CT]
- G06F 18/2148: {characterised by the process organisation or structure, e.g. boosting cascade}
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/08: Learning methods
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06N 3/0895: Weakly supervised learning, e.g. semi-supervised or self-supervised learning
- G06N 3/09: Supervised learning
- G06T 7/0012: {Biomedical image inspection}
- G06T 7/11: Region-based segmentation
- G06V 10/25: Determination of region of interest [ROI] or a volume of interest [VOI]
- G06V 10/758: {Involving statistics of pixels or of feature values, e.g. histogram matching}
- G06V 10/764: using classification, e.g. of video objects
- G06V 10/7747: {Organisation of the process, e.g. bagging or boosting}
- G06V 10/82: using neural networks
- G16H 30/20: for handling medical images, e.g. DICOM, HL7 or PACS
- G16H 30/40: for processing medical images, e.g. editing
- G06T 2207/10016: Video; Image sequence
- G06T 2207/10068: Endoscopic image
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/30004: Biomedical image processing
- G06T 2207/30096: Tumor; Lesion
- G06V 2201/03: Recognition of patterns in medical or anatomical images
- G16H: HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
- A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D)
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

There are provided a learning apparatus and a learning method that can facilitate creation of teaching data and prevent overtraining. A learning apparatus (10) includes a first database that stores a first image set in which a first image for learning and coordinate information for identifying a region larger than a region of interest included in the first image are associated with each other, and a second database that stores a second image set in which a second image for learning and second mask data for identifying a region of interest included in the second image are associated with each other. In a case of using the first image set to update a parameter of a CNN (32) (in a case of performing learning), a mask data creation unit (38) creates first mask data on the basis of the coordinate information for identifying the region larger than the region of interest. The first image and the second image are used as input images for the CNN (32), and the first mask data and the second mask data are used as teaching data to update the parameter of the CNN (32).

## Claims

### Claim 1

A learning apparatus comprising:
a memory that stores a first image set in which a first image for learning and coordinate information for identifying a region larger than a region of interest included in the first image are associated with each other and a second image set in which a second image for learning and second mask data for identifying a region of interest included in the second image on a per pixel basis or on a per block-of-pixels basis are associated with each other; and
a processor, configured to:
input the first image to a neural network and obtain a first feature map output from the neural network;
compare the first feature map with the coordinate information associated with the first image to calculate a first loss value;
update a plurality of parameters of the neural network on the basis of the first loss value;
obtain the second image and the second mask data from the second image set;
input the second image to the neural network and obtain a second feature map output from the neural network;
compare the second feature map with the second mask data associated with the second image to calculate a second loss value; and
update the plurality of parameters of the neural network on the basis of the second loss value.

### Claim 2 (depends on 1)

The learning apparatus according to claim 1, wherein the processor is further configured to:
create first mask data on the basis of the coordinate information associated with the first image, wherein
the first mask data corresponding to the first image is created before the first image is input to the neural network.

### Claim 3 (depends on 1)

The learning apparatus according to claim 1, wherein the coordinate information is information for identifying a rectangular region that includes the region of interest, and the first mask data is mask data in which the rectangular region is masked.

### Claim 4 (depends on 3)

The learning apparatus according to claim 3, wherein
a region smaller than the rectangular region is set within the rectangular region, and
the processor excludes a region that is within the rectangular region and outside the region smaller than the rectangular region from loss value calculation.

### Claim 5 (depends on 3)

The learning apparatus according to claim 3, wherein
a region smaller than the rectangular region is set within the rectangular region, and
the processor excludes the region smaller than the rectangular region from loss value calculation.

### Claim 6 (depends on 1)

The learning apparatus according to claim 1, wherein the coordinate information is information for identifying an elliptic region that includes the region of interest, and the first mask data is mask data in which the elliptic region is masked.

### Claim 7 (depends on 6)

The learning apparatus according to claim 6, wherein
a region smaller than the elliptic region is set within the elliptic region, and
the processor excludes a region that is within the elliptic region and outside the region smaller than the elliptic region from loss value calculation.

### Claim 8 (depends on 6)

The learning apparatus according to claim 6, wherein
a region smaller than the elliptic region is set within the elliptic region, and
the processor excludes the region smaller than the elliptic region from loss value calculation.

### Claim 9 (depends on 1)

The learning apparatus according to claim 1, wherein
the processor compares the first feature map smaller in size than the first image with the first mask data smaller in size than the first image to calculate the first loss value, and
the processor compares the second feature map smaller in size than the second image with the second mask data smaller in size than the second image to calculate the second loss value.

### Claim 10 (depends on 1)

The learning apparatus according to claim 1, wherein in the first image set, the first image that includes a blur is present.

### Claim 11 (depends on 1)

The learning apparatus according to claim 1, wherein the first image that constitutes the first image set and the second image that constitutes the second image set are medical images.

### Claim 12 (depends on 1)

The learning apparatus according to claim 1, wherein the first image that constitutes the first image set is an image extracted from a moving image.

### Claim 13 (depends on 1)

The learning apparatus according to claim 1, wherein the first image that constitutes the first image set and the second image that constitutes the second image set are images captured by an endoscope apparatus.

### Claim 14 (depends on 1)

The learning apparatus according to claim 1, wherein the second mask data is mask data in which a lesion region included in the second image is masked.

### Claim 15

A learning apparatus comprising:
a memory that stores a first image set in which a first image for learning and first mask data for identifying a region larger than a region of interest included in the first image are associated with each other and a second image set in which a second image for learning and second mask data for identifying a region of interest included in the second image on a per pixel basis or on a per block-of-pixels basis are associated with each other; and
a processor, configured to:
input the first image to a neural network and obtain a first feature map output from the neural network;
compare the first feature map with the first mask data associated with the first image to calculate a first loss value;
update a plurality of parameters of the neural network on the basis of the first loss value;
obtain the second image and the second mask data from the image set;
input the second image to the neural network and obtain a second feature map output from the neural network;
compare the second feature map with the second mask data associated with the second image to calculate a second loss value; and
update the plurality of parameters of the neural network on the basis of the second loss value.

### Claim 16 (depends on 1)

The learning apparatus according to claim 1, wherein each of the first mask data and the second mask data is mask data that is subjected to a pooling process at least once.

### Claim 17 (depends on 16)

The learning apparatus according to claim 16, wherein
the neural network has at least one pooling layer, and
each of the first mask data and the second mask data is mask data that is subjected to a pooling process corresponding to the pooling layer.

### Claim 18

A learning method comprising:
preparing a first image set in which a first image for learning and coordinate information for identifying a region larger than a region of interest included in the first image are associated with each other and a second image set in which a second image for learning and second mask data for identifying a region of interest included in the second image on a per pixel basis or on a per block-of-pixels basis are associated with each other;
inputting the first image to a neural network and obtaining a first feature map output from the neural network;
comparing the first feature map with the coordinate information associated with the first image to calculate a first loss value;
updating a plurality of parameters of the neural network on the basis of the first loss value;
obtaining the second image and the second mask data from the second image set;
inputting the second image to the neural network and obtaining a second feature map output from the neural network;
comparing the second feature map with the second mask data associated with the second image to calculate a second loss value; and
updating the plurality of parameters of the neural network on the basis of the second loss value.

### Claim 19

A learning method comprising:
preparing a first image set in which a first image for learning and first mask data for identifying a region larger than a region of interest included in the first image are associated with each other and a second image set in which a second image for learning and second mask data for identifying a region of interest included in the second image on a per pixel basis or on a per block-of-pixels basis are associated with each other;
inputting the first image to a neural network and obtaining a first feature map output from the neural network;
comparing the first feature map with the first mask data associated with the first image to calculate a first loss value;
updating a plurality of parameters of the neural network on the basis of the first loss value;
obtaining the second image and the second mask data from the image set;
updating a plurality of parameters of the neural network on the basis of the first loss value;
obtaining the second image and the second mask data from the image set;
inputting the second image to the neural network and obtaining a second feature map output from the neural network;
comparing the second feature map with the second mask data associated with the second image to calculate a second loss value; and
updating the plurality of parameters of the neural network on the basis of the second loss value.