# US 12,008,688 B2 · Method and electric device for processing breast Tomosynthesis images

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,008,688 B2
- Document ID: US12008688B2
- Status: Granted
- Kind: B2
- Application number: 17/446,457
- Filed: 2021-08-30
- Published: 2023-02-02
- Granted: 2024-06-11
- Earliest claimed: 2021-07-21
- Assignees: Institute for Information Industry
- Applicants: Institute for Information Industry
- Inventors: Kai-Hsuan Chan; Yao-Min Huang
- Web page: https://patentgrep.com/patent/US12008688B2

## CPC

- G06T 12/00: Tomographic reconstruction from projections
- A61B 6/025: {Tomosynthesis}
- A61B 6/502: {for diagnosis of breast, i.e. mammography}
- G06F 18/213: Feature extraction, e.g. by transforming the feature space; Summarisation; Mappings, e.g. subspace methods
- G06N 3/04: Architecture, e.g. interconnection topology
- G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU]
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/08: Learning methods
- G06T 7/0012: {Biomedical image inspection}
- G06T 11/003
- G16H 30/20: for handling medical images, e.g. DICOM, HL7 or PACS
- G16H 30/40: for processing medical images, e.g. editing
- G16H 50/20: for computer-aided diagnosis, e.g. based on medical expert systems
- G16H 50/70: for mining of medical data, e.g. analysing previous cases of other patients
- G06T 2207/10112: Digital tomosynthesis [DTS]
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/30068: Mammography; Breast
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D)
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G16H: HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA

## Abstract

A method for processing breast tomosynthesis images includes: obtaining multiple breast images, performing binarization operation and pooling operation to at least one marked breast image to obtain a lesion mask; training a convolutional neural network (CNN) according to the breast images and the lesion mask; obtaining multiple cropped images from the breast images of different layers according to a particular two-dimensional position, and inputting each cropped image into the CNN to obtain cropped heat maps; inputting the cropped heat maps into a recurrent neural network (RNN) in an order of the levels to obtain output heat maps, and combining the output heat maps to obtain a integrated heat map, and training the RNN according to the integrated heat map and the lesion mask.

## Claims

### Claim 1

A method for processing breast tomosynthesis images on an electrical device, the method comprising:
obtaining a plurality of breast images at a plurality of levels and performing a binarization operation and a pooling operation to at least one marked breast image of the breast images to generate at least one lesion mask, wherein the at least one lesion mask comprises a plurality of values to represent if there is a lesion or not;
training a convolutional neural network according to the breast images and the at least one lesion mask, wherein the convolutional neural network comprises a first branch and a second branch, the first branch is configured to output at least one value to indicate whether a lesion exists, the second branch is configured to output a heat map, and loss values between the heat map and the at least one lesion mask are configured to train the convolutional neural network;
obtaining a plurality of cropped images from the breast images at different ones of the levels according to a particular two-dimensional position, and inputting each of the cropped images into the convolutional neural network to obtain a plurality of cropped heat maps; and
inputting the cropped heat maps into a recurrent neural network in an order of the levels to obtain a plurality of output heat maps, combining the output heat maps to obtain a first integrated heat map, and training the recurrent neural network according to loss values between the first integrated heat map and the at least one lesion mask.

### Claim 2 (depends on 1)

The method of claim 1, wherein the convolutional neural network comprises a plurality of first convolution layers, the first branch and the second branch follow the first convolution layers, the first branch comprises a global average pooling layer and a fully connected layer, the second branch comprises a second convolution layer, and the second branch does not comprise the global average pooling layer and the fully connected layer.

### Claim 3 (depends on 2)

The method of claim 2, further comprising:
inputting a plurality of negative samples into the convolutional neural network, and retraining the convolutional neural network according to the negative samples that are determined to be positive.

### Claim 4 (depends on 2)

The method of claim 2, wherein the recurrent neural network is a long short-term memory (LSTM) network.

### Claim 5 (depends on 2)

The method of claim 2, wherein the first integrated heat map is an average of the output heat maps.

### Claim 6 (depends on 1)

The method of claim 1, wherein the first integrated heat map corresponds to a first angle, a second integrated heat map corresponds to a second angle, the method further comprises:
combining the first integrated heat map and the second integrated heat map to generate a lesion position.

### Claim 7

An electrical device, comprising:
a memory storing a plurality of instructions; and
a processor communicatively connected to the memory and is configured to execute the instructions,
wherein the processor is configured to obtain a plurality of breast images at a plurality of levels and perform a binarization operation and a pooling operation to at least one marked breast image of the breast images to generate at least one lesion mask, wherein the at least one lesion mask comprises a plurality of values to represent if there is a lesion or not,
wherein the processor is configured to train a convolutional neural network according to the breast images and the at least one lesion mask, wherein the convolutional neural network comprises a first branch and a second branch, the first branch is configured to output at least one value to indicate whether a lesion exists, the second branch is configured to output a heat map, and loss values between the heat map and the at least one lesion mask are configured to train the convolutional neural network,
wherein the processor is configured to obtain a plurality of cropped images from the breast images at different levels according to a particular two-dimensional position, and input each of the cropped images into the convolutional neural network to obtain a plurality of cropped heat maps,
wherein the processor is configured to input the cropped heat maps into a recurrent neural network in an order of the levels to obtain a plurality of output heat maps, combine the output heat maps to obtain a first integrated heat map, and train the recurrent neural network according to loss values between the first integrated heat map and the at least one lesion mask.

### Claim 8 (depends on 7)

The electrical device of claim 7, wherein the convolutional neural network comprises a plurality of first convolution layers, the first branch and the second branch follow the first convolution layers, the first branch comprises a global average pooling layer and a fully connected layer, the second branch comprises a second convolution layer, and the second branch does not comprise the global average pooling layer and the fully connected layer.

### Claim 9 (depends on 8)

The electrical device of claim 8, wherein the processor is configured to input a plurality of negative samples into the convolutional neural network, and retrain the convolutional neural network according to the negative samples that are determined to be positive.

### Claim 10 (depends on 8)

The electrical device of claim 8, wherein the recurrent neural network is a long short-term memory (LSTM) network.

### Claim 11 (depends on 8)

The electrical device of claim 8, wherein the first integrated heat map is an average of the output heat maps.

### Claim 12 (depends on 7)

The electrical device of claim 7, wherein the first integrated heat map corresponds to a first angle, a second integrated heat map corresponds to a second angle,
wherein the processor is configured to combine the first integrated heat map and the second integrated heat map to generate a lesion position.