# US 12,183,001 B2 · Content based image retrieval for lesion analysis

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,183,001 B2
- Document ID: US12183001B2
- Status: Granted
- Kind: B2
- Application number: 18/063,616
- Filed: 2022-12-08
- Published: 2023-04-06
- Granted: 2024-12-31
- Earliest claimed: 2017-11-22
- Assignees: Arterys Inc.
- Applicants: Arterys Inc.
- Inventors: Daniel Irving Golden; Fabien Rafael David Beckers; John Axerio-Cilies; Matthieu Le; Jesse Lieman-Sifry; Anitha Priya Krishnan; Sean Patrick Sall; Hok Kan Lau; Matthew Joseph Didonato; Robert George Newton; Torin Arni Taerum; Shek Bun Law; Carla Rosa Leibowitz; Angélique Sophie Calmon
- Web page: https://patentgrep.com/patent/US12183001B2

## CPC

- G06T 7/0012: {Biomedical image inspection}
- G16H 50/20: for computer-aided diagnosis, e.g. based on medical expert systems
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/08: Learning methods
- G06N 3/09: Supervised learning
- G06N 3/0985: Hyperparameter optimisation; Meta-learning; Learning-to-learn
- G06T 7/11: Region-based segmentation
- G06T 7/143: involving probabilistic approaches, e.g. Markov random field [MRF] modelling
- G06T 7/194: involving foreground-background segmentation
- G06V 10/82: using neural networks
- G16H 10/60: for patient-specific data, e.g. for electronic patient records
- G16H 20/10: relating to drugs or medications, e.g. for ensuring correct administration to patients
- G16H 20/40: relating to mechanical, radiation or invasive therapies, e.g. surgery, laser therapy, dialysis or acupuncture
- G16H 30/40: for processing medical images, e.g. editing
- G16H 50/70: for mining of medical data, e.g. analysing previous cases of other patients
- G16H 70/20: relating to practices or guidelines
- G06T 2207/10081: Computed x-ray tomography [CT]
- G06T 2207/10088: Magnetic resonance imaging [MRI]
- G06T 2207/20036: Morphological image processing
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/20152: Watershed segmentation
- G06T 2207/30056: Liver; Hepatic
- G06T 2207/30064: Lung nodule
- G06T 2207/30096: Tumor; Lesion
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G16H: HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

Computed Tomography (CT) and Magnetic Resonance Imaging (MRI) are commonly used to assess patients with known or suspected pathologies of the lungs and liver. In particular, identification and quantification of possibly malignant regions identified in these high-resolution images is essential for accurate and timely diagnosis. However, careful quantitative assessment of lung and liver lesions is tedious and time consuming. This disclosure describes an automated end-to-end pipeline for accurate lesion detection and segmentation.

## Claims

### Claim 1

A machine learning system, comprising:
at least one nontransitory processor-readable storage medium that stores at least one of processor-executable instructions or data, a trained convolutional neural network (CNN) model, a database that stores batches of image data that depict potentially cancerous lesions and their associated features, and query data for a query potentially cancerous lesion, the query data including image data and a set of associated features; and
at least one processor communicably coupled to the at least one nontransitory processor-readable storage medium, in operation, the at least one processor:
extracts features from the image data of the query potentially cancerous lesion utilizing the trained CNN model and includes the extracted features in the set of associated features of the query potentially cancerous lesion;
calculates a similarity of the query potentially cancerous lesion to one or more potentially cancerous lesions in the database based on at least a subset of the set of associated features of the query potentially cancerous lesion;
retrieves a plurality of potentially cancerous lesions from the database ranked by similarity to the query potentially cancerous lesion; and
causes presentation of the plurality of potentially cancerous lesions by performing at least one of:
causing displaying of the plurality of potentially cancerous lesions in stratified sections each corresponding to a different lesion type;
causing displaying of the plurality of potentially cancerous lesions in a multi-dimensional polar plot representing multiple dimensions of similarity; or
causing displaying of the plurality of potentially cancerous lesions with gaps separating one from another, wherein the gaps are calculated based on respective similarities between corresponding potentially cancerous lesions.

### Claim 2 (depends on 1)

The machine learning system of claim 1 wherein the at least one processor calculates the similarity based on a mathematical determination of distance between the features of the query potentially cancerous lesion and the features of the potentially cancerous lesions in the database.

### Claim 3 (depends on 2)

The machine learning system of claim 2 wherein the at least one processor calculates the similarity using at least one of the Euclidian distance, the Manhattan distance, the Minkowski distance, or the LP distance (with P>0), between the features of the query potentially cancerous lesion and the features of the potentially cancerous lesions in the database.

### Claim 4 (depends on 2)

The machine learning system of claim 2 wherein the determination of distance between the features further comprises using one or more user preferences at inference time or at training time.

### Claim 5 (depends on 2)

The machine learning system of claim 2 wherein the at least one processor selects similar lesions based on an approximate nearest neighbors algorithm.

### Claim 6

A method, comprising:
extracting features from image data of a query potentially cancerous lesion utilizing a trained convolutional neural network (CNN) model and including the extracted features in a set of associated features of the query potentially cancerous lesion;
calculating a similarity of the query potentially cancerous lesion to one or more potentially cancerous lesions in a database based on at least a subset of the set of associated features of the query potentially cancerous lesion;
retrieving a plurality of potentially cancerous lesions from the database ranked by similarity to the query potentially cancerous lesion; and
causing presentation of the plurality of potentially cancerous lesions by performing at least one of:
causing displaying of the plurality of potentially cancerous lesions in stratified sections each corresponding to a different lesion type;
causing displaying of the plurality of potentially cancerous lesions in a multi-dimensional polar plot representing multiple dimensions of similarity; or
causing displaying of the plurality of potentially cancerous lesions with gaps separating one from another, wherein the gaps are calculated based on respective similarities between corresponding potentially cancerous lesions.

### Claim 7 (depends on 6)

The method of claim 6, wherein the retrieved potentially cancerous lesions from the database are based on a stratification of one or more associated features.

### Claim 8 (depends on 7)

The method of claim 7 wherein the associated features used for stratification are based on histopathological assessment from biopsies of the potentially cancerous lesions.

### Claim 9 (depends on 8)

The method of claim 8 wherein at least one of the associated features used for stratification is whether or not the potentially cancerous lesion was determined to be cancerous.

### Claim 10 (depends on 6)

The method of claim 6 wherein retrieving the one or more potentially cancerous lesions from the database comprises retrieving only the potentially cancerous lesions that are most similar to the query potentially cancerous lesion or retrieving only the potentially cancerous lesions that are drawn from patients who are clinically or demographically similar to the patient from whom the query potentially cancerous lesion is drawn.

### Claim 11 (depends on 6)

The method of claim 6, further comprising causing presentation of the retrieved potentially cancerous lesions sorted by their similarity to the query potentially cancerous lesion to a user on a display.

### Claim 12 (depends on 6)

The method of claim 6, further comprising causing presentation of an indication of similarity or distance with respect to the query potentially cancerous lesion overlaid over or placed alongside the potentially cancerous lesions to a user on a display.

### Claim 13 (depends on 6)

The method of claim 6, further comprising causing presentation of lesions that are determined to be cancerous in a different section than lesions that are determined to be benign to a user on a display.

### Claim 14 (depends on 6)

The method of claim 6, further comprising causing presentation of an indication of whether the lesion has been determined to be benign or malignant using a color, symbol, or textual indication overlaid over or placed alongside the potentially cancerous lesion to a user on a display.

### Claim 15 (depends on 6)

The method of claim 6, further comprising causing presentation of results obtained from a biopsy for cancerous lesions using a color, symbol, or textual indication overlaid over or placed alongside the potentially cancerous lesion to a user on a display.

### Claim 16 (depends on 6)

The method of claim 6, further comprising causing presentation of features that are dissimilar between the query lesion and the retrieved lesions to a user on a display.

### Claim 17 (depends on 6)

The method of claim 6, further comprising classifying the query potentially cancerous lesion using a classification model based on associated features and diagnostic information for the K most similar retrieved potentially cancerous lesions that have associated diagnostic information.

### Claim 18 (depends on 17)

The method of claim 17 wherein the output of the classification is (a) whether the potentially cancerous lesions are benign or malignant or (b) a cancer subtype of the potentially cancerous lesions.

### Claim 19

A non-transitory computer-readable medium storing contents that, when executed by one or more processors, cause performance of actions comprising:
extracting features from image data of a query potentially cancerous lesion utilizing a trained convolutional neural network (CNN) model and including the extracted features in a set of associated features of the query potentially cancerous lesion;
calculating a similarity of the query potentially cancerous lesion to one or more potentially cancerous lesions in a database based on at least a subset of the set of associated features of the query potentially cancerous lesion;
retrieving a plurality of potentially cancerous lesions from the database ranked by similarity to the query potentially cancerous lesion; and
causing presentation of the plurality of potentially cancerous lesions by performing at least one of:
causing displaying of the plurality of potentially cancerous lesions in stratified sections each corresponding to a different lesion type;
causing displaying of the plurality of potentially cancerous lesions in a multi-dimensional polar plot representing multiple dimensions of similarity; or
causing displaying of the plurality of potentially cancerous lesions with gaps separating one from another, wherein the gaps are calculated based on respective similarities between corresponding potentially cancerous lesions.

### Claim 20 (depends on 19)

The computer-readable medium of claim 19 wherein calculating the similarity comprises using a regression model whose output is a similarity score for at least some of the potentially cancerous lesions in the database.

### Claim 21 (depends on 20)

The computer-readable medium of claim 20 wherein the features used as input to the regression model include one or more of image features extracted by a CNN or clinical features.

### Claim 22 (depends on 19)

The computer-readable medium of claim 19 wherein the actions further comprise modifying the calculation of the similarity based on a user's feedback regarding the relative similarity of the retrieved results.

### Claim 23 (depends on 22)

The computer-readable medium of claim 22 wherein the actions further comprise receiving input from the user to (a) indicate which results are similar or dissimilar, or (b) re-order results based on their preference for similarity.

### Claim 24 (depends on 19)

The computer-readable medium of claim 19 wherein the actions further comprise causing presentation of results of the retrieving as part of a software package whose indications for use include the facilitation of diagnosis planning and biopsy planning for lesions observed in radiological images.

### Claim 25 (depends on 24)

The computer-readable medium of claim 24 wherein the radiological images are acquired (a) for a purpose of cancer screening or (b) for a purpose of examining one or more potentially cancerous lesions that were known to exist or suspected to exist prior to acquisition of the radiological images.