# US 2021/0174938 A1 · Three-dimensional medical image generation

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0174938 A1
- Document ID: US20210174938A1
- Status: Published application
- Kind: A1
- Application number: 16/703,187
- Filed: 2019-12-04
- Published: 2021-06-10
- Earliest claimed: 2019-12-04
- Assignees: International Business Machines Corporation
- Applicants: International Business Machines Corporation
- Inventors: Sun Young Park; Dustin Michael Sargent; James G. Thompson
- Web page: https://patentgrep.com/patent/US20210174938A1

## CPC

- G16H 15/00: ICT specially adapted for medical reports, e.g. generation or transmission thereof
- G06F 18/24: Classification techniques
- G06F 18/2413: based on distances to training or reference patterns
- G06T 15/00: Three-dimensional [3D] image rendering
- G06V 10/764: using classification, e.g. of video objects
- G06V 10/82: using neural networks
- G16H 30/40: for processing medical images, e.g. editing
- G16H 50/20: for computer-aided diagnosis, e.g. based on medical expert systems
- G06T 2210/41: Medical
- G06V 2201/03: Recognition of patterns in medical or anatomical images
- G16H: HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

An embodiment of the invention may include a method, computer program product and computer system for three-dimensional medical image generation. The method, computer program product and computer system may include computing device which may receive a first three-dimensional medical image of a first patient from a first period of time, a two-dimensional medical image of the first patient from a second period of time and a plurality of three-dimensional medical images for a plurality of second patients. The computing device may input the three-dimensional medical image of the first patient, the two-dimensional medical image of the first patient and the plurality of three-dimensional medical images for a plurality of second patients into a generative adversarial network (GAN). The computing device may generate a synthetic three-dimensional medical image for the first patient based on the two-dimensional medical image from the second period of time utilizing the GAN.

## Claims

### Claim 1

A method for three-dimensional medical image generation, the method comprising:
receiving a first three-dimensional medical image, that was taken from a first patient at a first period of time;
receiving a two-dimensional medical image, that was taken from the first patient at a second period of time;
receiving a plurality of three-dimensional medical images for a plurality of second patients;
inputting the first three-dimensional medical image of the first patient, the two-dimensional medical image of the first patient and the plurality of three-dimensional medical images for the plurality of second patients into a generative adversarial network (GAN); and
generating a synthetic three-dimensional medical image for the first patient based on the two-dimensional medical image from the second period of time utilizing the GAN.

### Claim 2 (depends on 1)

The method of claim 1, further comprising:
analyzing the synthetic three-dimensional medical image of the first patient for one or more differences compared to the first three-dimensional medical image of the first patient; and
annotating the three-dimensional medical image of the first patient with indicators indicating the differences between the first three-dimensional medical image of the first patient and the synthetic three-dimensional image of the first patient.

### Claim 3 (depends on 2)

The method of claim 2, further comprising:
generating a report with a textual description of the differences; and
displaying at least one of the synthetic three-dimensional medical image of the first patient, the report, and the annotated three-dimensional medical image of the first patient to a user via a user interface.

### Claim 4 (depends on 1)

The method of claim 1, wherein the first three-dimensional medical image of the first patient is a digital breast tomosynthesis (DBT) image.

### Claim 5 (depends on 1)

The method of claim 1, wherein the two-dimensional medical image is a full-field digital mammography (FFDM) image.

### Claim 6 (depends on 1)

The method of claim 1, wherein the plurality of three-dimensional images for the plurality of second patients are digital breast tomosynthesis (DBT) images.

### Claim 7 (depends on 1)

The method of claim 1, wherein generating the synthetic three-dimensional medical image for the first patient based on the two-dimensional medical image from the second period of time utilizing the GAN further comprises:
inputting the synthetic three-dimensional medical image into a first discriminator of the GAN, wherein the first discriminator tries to discriminate between the synthetic three-dimensional medical image and the plurality of three-dimensional medical images from the plurality of second patients; and
inputting the synthetic three-dimensional medical image into a second discriminator of the GAN, wherein the second discriminator creates projected two-dimensional medical images corresponding to the synthetic three-dimensional medical image, wherein the second discriminator tries to discriminate between the projected two-dimensional medical images corresponding to the synthetic three-dimensional medical image and the two-dimensional medical image of the first patient from the second period of time.

### Claim 8

A computer program product for three-dimensional medical image generation, the computer program product comprising:
a computer-readable storage medium having program instructions embodied therewith, wherein the computer readable storage medium is not a transitory signal per se, wherein the program instructions are executable by a computer to cause the computer to perform a method, the method comprising:
receiving a first three-dimensional medical image, that was taken from a first patient at a first period of time;
receiving a two-dimensional medical image, that was taken from the first patient at a second period of time;
receiving a plurality of three-dimensional medical images for a plurality of second patients;
inputting the first three-dimensional medical image of the first patient, the two-dimensional medical image of the first patient and the plurality of three-dimensional medical images for the plurality of second patients into a generative adversarial network (GAN); and
generating a synthetic three-dimensional medical image for the first patient based on the two-dimensional medical image from the second period of time utilizing the GAN.

### Claim 9 (depends on 8)

The computer program product of claim 8, further comprising:
analyzing the synthetic three-dimensional medical image of the first patient for one or more differences compared to the first three-dimensional medical image of the first patient; and
annotating the three-dimensional medical image of the first patient with indicators indicating the differences between the first three-dimensional medical image of the first patient and the synthetic three-dimensional image of the first patient.

### Claim 10 (depends on 9)

The computer program product of claim 9, further comprising:
generating a report with a textual description of the differences; and
displaying at least one of the synthetic three-dimensional medical image of the first patient, the report, and the annotated three-dimensional medical image of the first patient to a user via a user interface.

### Claim 11 (depends on 8)

The computer program product of claim 8, wherein the first three-dimensional medical image of the first patient is a digital breast tomosynthesis (DBT) image.

### Claim 12 (depends on 8)

The computer program product of claim 8, wherein the two-dimensional medical image is a full-field digital mammography (FFDM) image.

### Claim 13 (depends on 8)

The computer program product of claim 8, wherein the plurality of three-dimensional images for the plurality of second patients are digital breast tomosynthesis (DBT) images.

### Claim 14 (depends on 8)

The computer program product of claim 8, wherein generating the synthetic three-dimensional medical image for the first patient based on the two-dimensional medical image from the second period of time utilizing the GAN further comprises:
inputting the synthetic three-dimensional medical image into a first discriminator of the GAN, wherein the first discriminator tries to discriminate between the synthetic three-dimensional medical image and the plurality of three-dimensional medical images from the plurality of second patients; and
inputting the synthetic three-dimensional medical image into a second discriminator of the GAN, wherein the second discriminator creates projected two-dimensional medical images corresponding to the synthetic three-dimensional medical image, wherein the second discriminator tries to discriminate between the projected two-dimensional medical images corresponding to the synthetic three-dimensional medical image and the two-dimensional medical image of the first patient from the second period of time.

### Claim 15

A system for three-dimensional medical image generation, the system comprising:
a computer system comprising: a processor, a computer readable storage medium, and program instructions stored on the computer readable storage medium that are executable by the processor to cause the computer system to:
receive a first three-dimensional medical image, that was taken from of a first patient at a first period of time;
receive a two-dimensional medical image, that was taken from the first patient at a second period of time;
receive a plurality of three-dimensional medical images for a plurality of second patients;
input the first three-dimensional medical image of the first patient, the two-dimensional medical image of the first patient and the plurality of three-dimensional medical images for the plurality of second patients into a generative adversarial network (GAN); and
generate a synthetic three-dimensional medical image for the first patient based on the two-dimensional medical image from the second period of time utilizing the GAN.

### Claim 16 (depends on 15)

The system of claim 15, further comprising program instructions to:
analyze the synthetic three-dimensional medical image of the first patient for one or more differences compared to the first three-dimensional medical image of the first patient; and
annotate the three-dimensional medical image of the first patient with indicators indicating the differences between the first three-dimensional medical image of the first patient and the synthetic three-dimensional image of the first patient.

### Claim 17 (depends on 16)

The system of claim 16, further comprising program instructions to:
generate a report with a textual description of the differences; and
display at least one of the second three-dimensional medical image of the first patient, the report, and the annotated three-dimensional medical image of the first patient to a user via a user interface.

### Claim 18 (depends on 15)

The system of claim 15, wherein the first three-dimensional medical image of the first patient and the plurality of three-dimensional images for the plurality of second patients are digital breast tomosynthesis (DBT) images.

### Claim 19 (depends on 15)

The system of claim 15, wherein the two-dimensional medical image is a full-field digital mammography (FFDM) image.

### Claim 20 (depends on 15)

The system of claim 15, wherein the program instructions to generate the synthetic three-dimensional medical image for the first patient based on the two-dimensional medical image from the second period of time utilizing the GAN further comprise program instructions to:
input the synthetic three-dimensional medical image into a first discriminator of the GAN, wherein the first discriminator tries to discriminate between the synthetic three-dimensional medical image and the plurality of three-dimensional medical images from the plurality of second patients; and
input the synthetic three-dimensional medical image into a second discriminator of the GAN, wherein the second discriminator creates projected two-dimensional medical images corresponding to the synthetic three-dimensional medical image, wherein the second discriminator tries to discriminate between the projected two-dimensional medical images corresponding to the synthetic three-dimensional medical image and the two-dimensional medical image of the first patient from the second period of time.
