# US 11,308,619 B2 · Evaluating a mammogram using a plurality of prior mammograms and deep learning algorithms

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,308,619 B2
- Document ID: US11308619B2
- Status: Granted
- Kind: B2
- Application number: 16/932,140
- Filed: 2020-07-17
- Published: 2022-01-20
- Granted: 2022-04-19
- Earliest claimed: 2020-07-17
- Assignees: International Business Machines Corporation
- Applicants: International Business Machines Corporation
- Inventors: Maria Victoria Sainz de Cea; David Richmond; Chao Song
- Web page: https://patentgrep.com/patent/US11308619B2

## CPC

- G06T 7/0014: {using an image reference approach}
- G06T 7/0016: {involving temporal comparison}
- G06F 18/214: Generating training patterns; Bootstrap methods, e.g. bagging or boosting
- G06F 18/24: Classification techniques
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU]
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/08: Learning methods
- G06N 3/09: Supervised learning
- G06N 5/01: Dynamic search techniques; Heuristics; Dynamic trees; Branch-and-bound
- G06N 20/20: Ensemble learning
- G06V 10/454: {Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]}
- G06V 10/82: using neural networks
- G16H 30/20: for handling medical images, e.g. DICOM, HL7 or PACS
- G16H 30/40: for processing medical images, e.g. editing
- G16H 50/20: for computer-aided diagnosis, e.g. based on medical expert systems
- G16H 50/70: for mining of medical data, e.g. analysing previous cases of other patients
- G06N 3/048: Activation functions
- G06T 2207/10116: X-ray image
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/30068: Mammography; Breast
- G06V 2201/03: Recognition of patterns in medical or anatomical images
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G16H: HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA

## Abstract

An approach for training, on a computer, one or more deep learning algorithms with a plurality of mammograms with known outcomes based, at least in part, on using a set of mammograms of each patient in the plurality of mammograms. The approach includes receiving a first set of mammograms of a first patient. The first set of mammograms includes an unevaluated mammogram of the first patient and a set of prior mammograms of the first patient. The approach includes the trained convolutional neural network extracting the set of features from each mammogram of the set of mammograms of the first patient. Furthermore, the approach includes using a second deep learning algorithm of the one or more deep learning algorithms to perform an evaluation of the unevaluated mammogram of the first patient based, at least in part, on the set of prior mammograms of the first patient.

## Claims

### Claim 1

A computer program product comprising:
one or more computer readable storage media and program instructions stored on the one or more computer readable storage media, the program instructions executable by a processor, the program instructions comprising instructions for:
extracting, by one or more computer processors, by a trained convolutional neural network, a set of features from each mammogram of a plurality of mammograms with known outcomes, further comprises training one of a random forest classifier or a modified long-short term memory algorithm, wherein training the modified long-short term algorithm includes providing one or more weighting factors associated with each mammogram of the set of mammograms of each patient from the plurality of patients;
receiving, by one or more computer processors, a first set of mammograms for a first patient wherein, the first set of mammograms of the first patient includes an unevaluated mammogram of the first patient and a set of prior mammograms of the first patient;
extracting, by one or more computer processors, by the trained convolutional neural network, the set of features from each mammogram of the first set of mammograms of the first patient; and
performing, by one or more computer processors, an evaluation of the unevaluated mammogram of the first patient using the trained convolutional neural network and a deep learning algorithm of the one or more trained deep learning algorithms based, at least in part, on the set of prior mammograms of the first patient.

### Claim 2 (depends on 1)

The computer program product of claim 1, further comprises providing an output of the evaluation of the unevaluated mammogram of the first patient wherein, the output includes one or more of a graphical output, a numerical output, a written output, or a binary outcome.

### Claim 3 (depends on 2)

The computer program product of claim 2, wherein the binary outcome includes one of normal or abnormal.

### Claim 4 (depends on 1)

The computer program product of claim 1, wherein an image quality of each mammogram of the set of mammograms of each patient is a weighting factor of the one or more weighting factors associated with each mammogram of the set of mammograms of each patient.

### Claim 5 (depends on 1)

The computer program product of claim 1, wherein training the modified long-short term algorithm includes a weighting factor associated with a time elapsed from a current mammogram being evaluated to each prior mammogram in the set of mammograms of each patient wherein, a more recent mammogram in the set of mammograms with less time elapsed from the current mammogram being evaluated is given more weight than an older mammogram.

### Claim 6 (depends on 1)

The computer program product of claim 1, wherein training the training the random forest classifier includes using the trained convolutional neural network, further comprises:
receiving, by one or more computer processors, the first set of mammograms of the first patient wherein, the first set of mammograms of the first patient includes the unevaluated mammogram of the first patient and the set of prior mammograms of the first patient;
extracting, by one or more computer processors, by the trained convolutional neural network, the set of features from each mammogram of the first set of mammograms of the first patient; and
performing, by one or more computer processors, the evaluation of the unevaluated mammogram of the first patient using the trained random forest classifier based, at least in part, on the set of prior mammograms of the first patient.

### Claim 7 (depends on 1)

The computer program product of claim 1, wherein training the random forest classifier further comprises:
receiving, by one or more computer processors, a user input of a number of mammograms needed for the set of mammograms of each patient of the plurality of patients in the training of the random forest classifier;
creating, by one or more computer processors, a feature vector concatenating each feature of the set of features from each mammogram of the plurality of mammograms with the known outcomes;
determining, by one or more computer processors, that one or more mammograms of the number of mammograms needed for the set of mammograms of each patient of the plurality of patients in the training of the random forest classifier is missing; and
inputting, by one or more computer processors, zeros for each feature of the set of features associated with the one or more missing mammograms.

### Claim 8

A computer system comprising:
one or more computer processors;
one or more computer readable storage media;
program instructions stored on the one or more computer readable storage media for execution by at least one of the one or more processors, the program instructions comprising instructions to perform:
extracting, by one or more computer processors, by a trained convolutional neural network, a set of features from each mammogram of a plurality of mammograms with known outcomes:
training, by one or more computer processors, one or more deep learning algorithms based, at least in part, on using the set of features from each mammogram of a set of mammograms of each patient of a plurality of patients from the plurality of mammograms with the known outcomes, further comprises training one of a random forest classifier or a modified long-short term memory algorithm, wherein training the modified long-short term algorithm includes providing one or more weighting factors associated with each mammogram of the set of mammograms of each patient;
receiving, by one or more computer processors, a first set of mammograms for a first patient wherein, the first set of mammograms includes an unevaluated mammogram of the first patient and a set of prior mammograms of the first patient;
extracting, by one or more computer processors, by the trained convolutional neural network, the set of features from each mammogram of the first set of mammograms of the first patient; and
performing, by one or more computer processors, an evaluation of the unevaluated mammogram of the first patient using the trained convolutional neural network and a trained deep learning algorithm of the one or more trained deep learning algorithms based, at least in part, on the set of prior mammograms of the first patient.

### Claim 9 (depends on 8)

The computer system of claim 8, wherein an image quality of each mammogram of the set of mammograms of each patient is a weighting factor of the one or more weighting factors associated with each mammogram of the set of mammograms of each patient.

### Claim 10 (depends on 8)

The computer system of claim 8, wherein training the random forest classifier further comprises:
receiving, by one or more computer processors, a user input of a number of mammograms needed for the set of mammograms of each patient of the plurality of patients in the training of the random forest classifier;
creating, by one or more computer processors, a feature vector concatenating each feature of the set of features from each mammogram of the plurality of mammograms with the known outcomes;
determining, by one or more computer processors, that one or more mammograms of the number of mammograms needed for the set of mammograms of each patient of the plurality of patients in the training of the random forest classifier is missing; and
inputting, by one or more computer processors, zeros for each feature of the set of features associated with the one or more missing mammograms.

### Claim 11 (depends on 8)

The computer system of claim 8, wherein the weighting factor is associated with a time elapsed from a current mammogram being evaluated to each prior mammogram in the set of mammograms of each patient wherein, a more recent mammogram in the set of mammograms with less time elapsed from the current mammogram being evaluated is given more weight than an older mammogram.

### Claim 12 (depends on 1)

The computer program product of claim 1, wherein training the modified long-short term algorithm can be modified to detect asymmetry by modifications to the program to include mammogram views from both a left side and a right side of each patient of the plurality of patients.