# US 2022/0036612 A1 · Systems and methods for generating summary medical images

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2022/0036612 A1
- Document ID: US20220036612A1
- Status: Published application
- Kind: A1
- Application number: 16/942,805
- Filed: 2020-07-30
- Published: 2022-02-03
- Earliest claimed: 2020-07-30
- Assignees: International Business Machines Corporation
- Applicants: International Business Machines Corporation
- Inventors: FLORA GILBOA-SOLOMON; Aviad Zlotnick
- Web page: https://patentgrep.com/patent/US20220036612A1

## CPC

- G06V 10/761: {Proximity, similarity or dissimilarity measures}
- G06F 18/22: Matching criteria, e.g. proximity measures
- G06N 3/045: Combinations of networks
- G06N 3/0455: Auto-encoder networks; Encoder-decoder networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/08: Learning methods
- G06N 3/09: Supervised learning
- G06N 20/00: Machine learning
- G06T 11/60: Creating or editing images; Combining images with text
- G06V 10/40: Extraction of image or video features
- G06V 10/803: {of input or preprocessed data}
- G06V 10/82: using neural networks
- G16H 30/40: for processing medical images, e.g. editing
- G16H 50/20: for computer-aided diagnosis, e.g. based on medical expert systems
- G06T 2210/41: Medical
- G06V 2201/03: Recognition of patterns in medical or anatomical images
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G16H: HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA

## Abstract

There is provided a computer implemented method for generating summary images from 3D medical images, comprising: receiving a 3D medical image, dividing the 3D medical images into a sequence of a plurality 2D images, computing a similarity dataset indicative of an amount of similarity between each pair of the plurality of 2D images, segmenting the similarity dataset into a plurality of slabs by minimizing the amount of similarity between consecutive slabs and maximizing the amount of similarity within each slab, aggregating, for each respective slab, the plurality of 2D images into a respective summary image, and presenting on a display, the respective summary image.

## Claims

### Claim 1

A computer implemented method for generating summary images from 3D medical images, comprising:
receiving a 3D medical image;
dividing the 3D medical images into a sequence of a plurality 2D images;
computing a similarity dataset indicative of an amount of similarity between each pair of the plurality of 2D images;
segmenting the similarity dataset into a plurality of slabs by minimizing the amount of similarity between consecutive slabs and maximizing the amount of similarity within each slab;
aggregating, for each respective slab, the plurality of 2D images into a respective summary image; and
presenting on a display, the respective summary image.

### Claim 2 (depends on 1)

The method of claim 1, wherein aggregating comprises computing, for each respective slab, a maximum intensity projection (MIP) from the plurality of 2D images, wherein the summary image comprises the MIP.

### Claim 3 (depends on 1)

The method of claim 1, further comprising:
selecting a sub-sample set of at least one 2D image from each respective slab, wherein the sub-sample includes fewer 2D images than the respective slab;
inputting each 2D image of each respective sub-sample set of each respective slab into a 2D machine learning model trained on a training dataset of a plurality of sample 2D images and corresponding at least one visual feature depicted therein indicative of ground truth; and
obtaining a score indicative of the presence of at least one finding depicted in each 2D image as an outcome of the 2D machine learning model.

### Claim 4 (depends on 3)

The method of claim 3, wherein the sub-sample set includes at least one boundary 2D image that is sequentially adjacent to another slab.

### Claim 5 (depends on 3)

The method of claim 3, wherein the 2D machine learning model includes a simple deep neural network (DNN) and excludes a feature pyramid that processes inputted 3D images in a plurality of resolutions and with a plurality of box shapes.

### Claim 6 (depends on 1)

The method of claim 1, wherein the similarity dataset is a matrix of size N×N wherein N denotes a number of the plurality of 2D images.

### Claim 7 (depends on 1)

The method of claim 1, further comprising:
inputting each 2D image into a neural network trained to analyze a target 2D image and output an indication of a target visual feature being depicted within the target 2D image;
extracting, for each 2D image, a feature vector from the neural network;
wherein the amount of similarity is computed for the feature vectors of each pair.

### Claim 8 (depends on 7)

The method of claim 7, wherein the amount of similarity is computed for the feature vectors of each pair using a cosine similarity.

### Claim 9 (depends on 7)

The method of claim 7, wherein the feature vector is selected from the group consisting of: embeddings obtained from hidden layers of the neural network, an output of an autoencoder implementation of the neural network, and the feature vector is an output of the neural network.

### Claim 10 (depends on 9)

The method of claim 9, further comprising selecting the neural network from a plurality of neural networks each trained to output an indication of a different visual feature, according to the visual feature, wherein the user is viewing the 3D medical image to search for the visual feature.

### Claim 11 (depends on 1)

The method of claim 1, wherein the plurality of slabs are computed by inputting the sequential 2D images into a video scene analysis process that divides a video into scenes of frames, the 2D images corresponding to frames of the video and the scenes corresponding to slabs.

### Claim 12 (depends on 1)

The method of claim 1, wherein the 3D medical image is selected from the group consisting of: CT, MRI, breast tomography, 3D ultrasound, 3D nuclear imaging, and PET.

### Claim 13 (depends on 1)

The method of claim 1, wherein the 3D medical images are divided into a sequence of a plurality 2D images according to a slice orientation and/or a slice thickness defined by the user viewing the plurality of 2D images and corresponds to the slice orientation and/or slice thickness when the user views the plurality of 2D images.

### Claim 14

A computer implemented method for generating summary images from sequentially acquired real time 2D medical images, comprising:
in a plurality of iterations:
receiving a 2D medical image captured in real time;
computing an amount of similarity between the 2D medical image and at least one previously obtained sequence of 2D medical images;
in response to the amount of similarity being below a threshold, determining that the 2D medical image represents an end of a slab of a plurality of sequentially obtained 2D medical images;
aggregating the plurality of sequentially obtained 2D images of the slab into a summary image; and
presenting the summary image on a display.

### Claim 15 (depends on 14)

The method of claim 14, wherein aggregating comprises computing, for each respective slab, a maximum intensity projection (MIP) from the plurality of 2D images, wherein the summary image comprises the MIP.

### Claim 16 (depends on 14)

The method of claim 14, further comprising:
inputting at least one respective summary image into a summary machine learning model trained on a training dataset of a plurality of sample summary images and corresponding at least one visual feature depicted therein indicative of ground truth; and
obtaining at least one visual finding depicted in the summary image as an outcome of the summary machine learning model.

### Claim 17 (depends on 14)

The method of claim 14, further comprising:
in response to the amount of similarity being above a threshold indicating that the 2D medical image does not represent the end of the slab, including the 2D medical image as part of the slab, and performing another iteration by receiving another 2D medical image.

### Claim 18 (depends on 14)

The method of claim 14, wherein the 2D medical image and the sequence of 2D images are captured by an imaging device selected from the group consisting of: colonoscope, endoscope, bronchoscope, and 2D ultrasound.

### Claim 19 (depends on 14)

The method of claim 14, further comprising:
inputting the 2D medical image into a neural network trained to analyze a target 2D medical image and output an indication of a target visual feature being depicted within the target 2D medical image; and
extracting, for the 2D medical image, a feature vector from the neural network;
wherein computing the amount of similarity comprises computing the amount of similarity between the feature vector of the 2D medical image and feature vectors of the plurality of sequentially obtained 2D medical images.

### Claim 20

A computer implemented method for generating summary images from 3D medical images, comprising:
receiving a 3D medical image;
dividing the 3D medical images into a sequence of a plurality 2D images;
computing a similarity dataset indicative of an amount of similarity between each pair of the plurality of 2D images;
segmenting the similarity dataset into a plurality of slabs by minimizing the amount of similarity between consecutive slabs and maximizing the amount of similarity within each slab;
aggregating, for each respective slab, the plurality of 2D images into a respective summary image;
inputting each respective summary image into a summary machine learning model trained on a training dataset of a plurality of sample summary images and corresponding at least one finding depicted therein indicative of ground truth; and
obtaining a score indicative of at least one finding depicted in the summary image as an outcome of the summary machine learning model.
