# US 10,368,846 B2 · Classifying hormone receptor status of malignant tumorous tissue from breast thermographic images

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,368,846 B2
- Document ID: US10368846B2
- Status: Granted
- Kind: B2
- Application number: 15/636,739
- Filed: 2017-06-29
- Published: 2018-01-04
- Granted: 2019-08-06
- Earliest claimed: 2016-06-29
- Applicants: Niramai Health Analytix Pvt. Ltd.
- Inventors: Krithika Venkataramani; Siva Teja Kakileti; Himanshu J. Madhu
- Web page: https://patentgrep.com/patent/US10368846B2

## CPC

- A61B 10/0041: {Detection of breast cancer (mammography by transillumination A61B5/0091, by X-rays A61B6/502, by ultrasonic means A61B8/0825)}
- A61B 5/015: {By temperature mapping of body part}
- A61B 5/4312: {Breast evaluation or disorder diagnosis (optical mammography A61B5/0091)}
- A61B 5/489: {Blood vessels}
- C07K 16/2869: {against hormone receptors (for antibodies against neuromediator receptors C07K16/286)}
- C12Q 1/6886: for cancer (immunoassay for cancer G01N33/575)
- G06T 7/0012: {Biomedical image inspection}
- G06T 7/11: Region-based segmentation
- A61B 5/00: Measuring for diagnostic purposes (radiation diagnosis A61B6/00; diagnosis by ultrasonic, sonic or infrasonic waves A61B8/00); Identification of persons
- A61B 2576/02: specially adapted for a particular organ or body part
- G01N 33/57515: of the breast
- G06T 2207/10048: Infrared image
- G06T 2207/30068: Mammography; Breast
- G16H 30/40: for processing medical images, e.g. editing
- A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D)
- C07K: PEPTIDES (peptides in foodstuffs A23; obtaining protein compositions for foodstuffs, working-up proteins for foodstuffs A23J; preparations for medicinal purposes A61K; peptides containing beta-lactam rings C07D; cyclic dipeptides not having in their molecule any other peptide link than those which form their ring, e.g. piperazine-2,5-diones, C07D; ergot alkaloids of the cyclic peptide type C07D519/02; macromolecular compounds having statistically distributed amino acid units in their molecules, i.e. when the preparation does not provide for a specific; but for a random sequence of the amino acid units, homopolyamides and block copolyamides derived from amino acids C08G69/00; macromolecular products derived from proteins C08H1/00; preparation of glue or gelatine C09H; single cell proteins, enzymes C12N; genetic engineering processes for obtaining peptides C12N15/00; compositions for measuring or testing processes involving enzymes C12Q; investigation or analysis of biological material G01N33/00)
- C12Q: MEASURING OR TESTING PROCESSES INVOLVING ENZYMES, NUCLEIC ACIDS OR MICROORGANISMS (immunoassay G01N33/53); COMPOSITIONS OR TEST PAPERS THEREFOR; PROCESSES OF PREPARING SUCH COMPOSITIONS; CONDITION-RESPONSIVE CONTROL IN MICROBIOLOGICAL OR ENZYMOLOGICAL PROCESSES
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G01N: INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES (measuring or testing apparatus or processes other than immunoassay, involving enzymes or microorganisms C12M, C12Q)
- G16H: HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA

## Abstract

What is disclosed is a system and method for classifying the hormone receptor status of malignant tumorous tissue identified in a thermographic image of a breast. One embodiment of the present method involves the following. First, receive thermographic image(s) of both breasts of a patient with a malignant tumorous region and analyzing the image(s) to define a boundary contour of the breast. Then, the breast regions are segment into regions of elevated temperature. A function of first probability mass function Q, ƒ(Q), is determined based on temperatures of pixels within a first segmented region. A function of the second probability mass function P, ƒ(P), is determined based on temperatures of pixels within a second region. A distance measure between the two functions ƒ(Q) and ƒ(P) is calculated and provided to a classifier trained to classify the malignant tissue as hormone receptor positive, and negative otherwise, based on the distance measure.

## Claims

### Claim 1

A method for classifying the hormone receptor status of malignant tumorous tissue identified in a thermographic image of a breast, the method comprising:
receiving thermographic image(s) of both breasts of a patient with a malignant tumorous region;
segmenting the breast regions into regions of elevated temperature and otherwise; and
analyzing temperature distribution of the thermographic image of the whole breast and the elevated temperature regions to classify the malignant tumorous tissue as being hormone receptor positive, and hormone receptive negative otherwise, wherein said analyzing the elevated temperature regions comprises comparison of the temperature distribution of the segmented regions, wherein said analyzing for the entire breast region comprises extracting a gray-level co-occurrence matrix (GLCM) and other textural features and using the co-occurrence features of the two breasts for classifying malignant tumorous tissue,
where a first segmented region is the elevated temperature region corresponding to the malignant tumorous tissue region; a second segmented region is the non-elevated temperature region in the same breast, wherein a distance measure representing a similarity between the first segmented region and the second segmented region is determined, and wherein the distance measure is provided to a classifier system to classify the hormone receptor class as either positive or negative.

### Claim 2 (depends on 1)

The method of claim 1,
wherein the first segmented region is represented by probability mass function P based on temperature values of pixels within the first segmented region, wherein the second segmented region is represented by probability mass function Q based on temperature values of pixels within the second segmented region, wherein a distance measure between the two functions P and Q is determined, and wherein the distance measure is provided to the classifier system to classify the hormone receptor class as either positive or negative.

### Claim 3 (depends on 2)

The method of claim 2, wherein the first segmented region is the elevated temperature region corresponding to the malignant tumorous tissue region; the second segmented region is the non-elevated temperature region in the same breast; the function of the first probability mass function is the normalization after subtracting the mean temperature of a third elevated temperature region in the contralateral breast to the malignant tumorous breast; the function of the second probability mass function is the normalization after subtracting the mean temperature of a fourth non-elevated temperature region in the contralateral breast to the malignant tumorous breast.

### Claim 4 (depends on 2)

The method of claim 2, wherein the first segmented region is the elevated temperature region corresponding to the malignant tumorous tissue region; the second segmented region is the elevated temperature region in the contralateral breast; ƒ(Q)=ΣiiQ(i); ƒ(P)=ΣiiP(i)=μp; the distance function is the mean squared distance between ƒ(Q) and ƒ(P) given by Σi∥i−μp∥2Q(i), where the ƒ(Q) is a function of first probability mass function Q, the (P) is a function of second probability mass function P, the i is a iterator variable, the Q(i) is a iterator variable of the first probability mass function Q, the P(i) is a iterator variable of the second probability mass function Q, and the μp is a mean of P.

### Claim 5 (depends on 2)

The method of claim 2, wherein the first segmented region is the non-elevated temperature region corresponding to the same breast having the malignant tumorous tissue; the second segmented region is the elevated temperature region in the contralateral breast; ƒ(Q)=ΣiiQ(i); ƒ(P)=ΣiiP(i)=μp; the distance function is the mean squared distance between ƒ(Q) and ƒ (P) given by Σi∥i−μp∥2Q(i), where the ƒ(Q) is a function of first probability mass function Q, the ƒ(P) is a function of second probability mass function P, the i is a iterator variable, the Q(i) is a iterator variable of the first probability mass function Q, the P(i) is a iterator variable of the second probability mass function Q, and the μp is a mean of P.

### Claim 6 (depends on 2)

The method of claim 2, where Q is obtained by normalizing corresponding histograms of pixels comprising the segmented region.

### Claim 7 (depends on 2)

The method of claim 2, wherein the distance function can be the Jensen Shannon distance, the Kullback Liebler distance, mutual information, or any standard defined distance function between two probability mass functions.

### Claim 8 (depends on 1)

The method of claim 1, further comprising:
receiving a thermographic image of a contra-lateral breast of the patient;
analyzing the thermographic image to identify a hot spot in the contra-lateral breast, the hot spot comprising a patch of pixels with an elevated temperature with respect to surrounding tissue;
determining a second distance measure between P−μ1 and Q−μ2, where μ1 is a mean temperature of the hot spot identified in the contra-lateral breast, and μ2 is the mean temperature of the surrounding tissue in the contra-lateral breast, where Q is the PMF of the hotspot related to the malignant tumorous tissue, and P is the PMF of the normal region in the breast having the malignant tumor; and
providing the second distance measure the classifier system.

### Claim 9 (depends on 1)

The method of claim 1, wherein the classifier system comprises any of: Support Vector Machine, a neural network, a Bayesian network, a Logistic regression, Naïve Bayes, Randomized Forests, Decision Trees and Boosted Decision Trees, K-nearest neighbor, a Restricted Boltzmann Machine, and a hybrid system comprising any combination hereof.

### Claim 10 (depends on 1)

The method of claim 1, wherein, in response to more than one hot spot being identified in either the left or right breast, further comprising selecting one of the hot spots for comparison purposes.

### Claim 11 (depends on 1)

The method of claim 1, further comprising updating a training set use to train the classifier system with the classified hot spot.

### Claim 12 (depends on 1)

The method of claim 1, further comprising communicating the classification to any of: a storage device, a display device, and a remote device over a network.

### Claim 13

A system for classifying the hormone receptor status of malignant tumorous tissue identified in a thermographic image of a breast, the system comprising:
a storage device containing a classifier system trained to classify a hot spot as one of: hormone receptor positive (HR+), and hormone receptive negative (HR−); and
a processor in communication with the storage device, the processor retrieving executing machine readable program instructions which, when executed by the processor, enable the processor to:
receive thermographic image(s) of both breasts of a patient with a malignant tumorous region;
analyze the image(s) to define a boundary contour of the breast in the image(s);
segment the breast regions into regions of elevated temperature and otherwise; and
analyze temperature distribution of the thermographic image of the whole breast and the elevated temperature regions to classify the malignant tumorous tissue as being hormone receptor positive, and hormone receptive negative otherwise, wherein said analyzing the elevated temperature regions comprises comparison of the temperature distribution of the segmented regions, wherein said analyzing for the entire breast region comprises extracting a gray-level co-occurrence matrix (GLCM) and other textural features and using the co-occurrence features of the two breasts for classifying malignant tumorous tissue,
where a first segmented region is the elevated temperature region corresponding to the malignant tumorous tissue region; a second segmented region is the non-elevated temperature region in the same breast, wherein a distance measure representing a similarity between the first segmented region and the second segmented region is determined, and wherein the distance measure is provided to a classifier system to classify the hormone receptor class as either positive or negative.

### Claim 14 (depends on 13)

The system of claim 13, wherein the first segmented region is represented by probability mass function P based on temperature values of pixels within the first segmented region, wherein the second segmented region is represented by probability mass function Q based on temperature values of pixels within the second segmented region, wherein the processor is configured to determine a distance measure between the two functions P and Q, and wherein the distance measure is provided to the classifier system to classify the hormone receptor class as either positive or negative.

### Claim 15 (depends on 14)

The system of claim 14, wherein the first segmented region is the elevated temperature region corresponding to the malignant tumorous tissue region; the second segmented region is the non-elevated temperature region in the same breast; the function of the first probability mass function is the normalization after subtracting the mean temperature of a third elevated temperature region in the contralateral breast to the malignant tumorous breast; the function of the second probability mass function is the normalization after subtracting the mean temperature of a fourth non-elevated temperature region in the contralateral breast to the malignant tumorous breast.

### Claim 16 (depends on 14)

The system of claim 14, where Q is obtained by normalizing corresponding histograms of pixels comprising the segmented region.

### Claim 17 (depends on 14)

The system of claim 14, wherein the distance function can be the Jensen Shannon distance, the Kullback Liebler distance, mutual information, or any standard defined distance function between two probability mass functions.

### Claim 18 (depends on 13)

The system of claim 13, wherein the first segmented region is the elevated temperature region corresponding to the malignant tumorous tissue region; the second segmented region is the elevated temperature region in the contralateral breast; ƒ(Q)=ΣiiQ(i); ƒ(P)=ΣiiP(i)=μp; the distance function is the mean squared distance between ƒ(Q) and ƒ(P) given by Σi∥i−μp∥2Q(i), where the ƒ(Q) is a function of first probability mass function Q the ƒ(P) is a function of second probability mass function P, the i is a iterator variable, the Q(i) is a iterator variable of the first probability mass function Q, the P(i) is a iterator variable of the second probability mass function Q and then is a mean of P.

### Claim 19 (depends on 13)

The system of claim 13, wherein the first segmented region is the non-elevated temperature region corresponding to the same breast having the malignant tumorous tissue; the second segmented region is the elevated temperature region in the contralateral breast; ƒ(Q)=ΣiiQ(i); ƒ(P)=ΣiiP(i)=μp; the distance function is the mean squared distance between ƒ(Q) and ƒ(P) given by Σi∥i−μp∥2Q(i), where the AO is a function of first probability mass function Q, the ƒ(P) is a function of second probability mass function P, the i is a iterator variable, the Q(i) is a iterator variable of the first probability mass function Q, the P(i) is a iterator variable of the second probability mass function Q, and the μp is a mean of P.

### Claim 20 (depends on 13)

The system of claim 13, wherein the processor is further configured to:
receive a thermographic image of a contra-lateral breast of the patient;
analyze the thermographic image to identify a hot spot in the contra-lateral breast, the hot spot comprising a patch of pixels with an elevated temperature with respect to surrounding tissue;
determine a second distance measure between P−μ1 and Q−μ2, where μ1 is a mean temperature of the hot spot identified in the contra-lateral breast, and μ2 is the mean temperature of the surrounding tissue in the contra-lateral breast, where Q is the PMF of the hotspot related to the malignant tumorous tissue, and P is the PMF of the normal region in the breast having the malignant tumor; and
provide the second distance measure to the classifier system.

### Claim 21 (depends on 13)

The system of claim 13, wherein the classifier system comprises any of: Support Vector Machine, a neural network, a Bayesian network, a Logistic regression, Naïve Bayes, Randomized Forests, Decision Trees and Boosted Decision Trees, K-nearest neighbor, a Restricted Boltzmann Machine, and a hybrid system comprising any combination hereof.

### Claim 22 (depends on 13)

The system of claim 13, wherein, in response to more than one hot spot being identified in either the left or right breast, further comprising selecting one of the hot spots for comparison purposes.

### Claim 23 (depends on 13)

The system of claim 13, wherein the processor is further configured to update a training set use to train the classifier system with the classified hot spot.

### Claim 24 (depends on 13)

The system of claim 13, wherein the processor is further configured to communicate the classification to any of: a display device, and a remote device over a network.

### Claim 25

A method for classifying the hormone receptor status of malignant tumorous tissue identified in a thermographic image of a breast, the method comprising:
receiving thermographic image(s) of both breasts of a patient with a malignant tumorous region;
analyzing the image(s) to define a boundary contour of the breast in the image(s);
generating a temperature map by subtracting the pixel temperatures of the malignant breast with the mean temperature of the contra-lateral breast;
computing a Run-Length-Matrix from the above temperature map, after quantizing the temperatures into L bins;
computing textural non-uniformity features from the Run Length Matrix; and
classifying the malignant tumorous tissue as hormone receptor negative if the non-uniformity feature is higher or hormone receptor positive otherwise, using a classifier system.

### Claim 26 (depends on 25)

The method of claim 25, wherein additional energy features and other textural features from the temperature map are used as additional inputs to the classifier system to determine whether the malignant tumorous tissue is hormone receptor positive or negative.

### Claim 27

A method for classifying the hormone receptor status of malignant tumorous tissue identified in a thermographic image of a breast, the method comprising:
receiving thermographic image(s) of both breasts of a patient with a malignant tumorous region;
analyzing the image(s) to define a boundary contour of the breast in the image(s);
segmenting the breast regions into regions of elevated temperature and otherwise;
determining the ratio between the area of elevated temperature region(s) in the malignant tumorous breast and the area of the elevated temperature region(s) in the contralateral breast, or vice-versa by analyzing the elevated temperature regions, wherein said analysis of the elevated temperature regions comprises the step of (i) comparison of temperature distribution of the segmented regions and (ii) extracting a gray-level co-occurrence matrix (GLCM) and other textural features and using the co-occurrence features of the two breasts for classifying malignant tumorous tissue;
providing the above area ratio and co-occurrence of features to the classifier system: and
classifying the malignant tumorous region as hormone receptor positive if the ratio is higher or hormone receptor negative if the ratio is zero, using a classifier system.