# US 11,908,047 B2 · Generating synthetic x-ray images and object annotations from CT scans for augmenting x-ray abnormality assessment systems

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,908,047 B2
- Document ID: US11908047B2
- Status: Granted
- Kind: B2
- Application number: 17/249,735
- Filed: 2021-03-11
- Published: 2022-09-15
- Granted: 2024-02-20
- Earliest claimed: 2021-03-11
- Assignees: Siemens Healthineers AG
- Applicants: Siemens Healthcare GmbH
- Inventors: Boris Mailhe; Florin-Cristian Ghesu; Siqi Liu; Sasa Grbic; Sebastian Vogt; Dorin Comaniciu; Awais Mansoor; Sebastien Piat; Steffen Kappler; Ludwig Ritschl
- Web page: https://patentgrep.com/patent/US11908047B2

## CPC

- G06T 12/30: Image post-processing, e.g. metal artefact correction
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06T 3/4046: using neural networks
- G06T 3/4053: based on super-resolution, i.e. the output image resolution being higher than the sensor resolution
- G06T 7/0012: {Biomedical image inspection}
- G06T 7/11: Region-based segmentation
- G16H 50/20: for computer-aided diagnosis, e.g. based on medical expert systems
- G16H 50/50: for simulation or modelling of medical disorders
- G16H 50/70: for mining of medical data, e.g. analysing previous cases of other patients
- G06N 3/045: Combinations of networks
- G06N 3/0475: Generative networks
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06N 3/094: Adversarial learning
- G06T 2207/10081: Computed x-ray tomography [CT]
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2211/408: Dual energy
- G06T 2211/441: AI-based methods, deep learning or artificial neural networks
- G06T 2211/464: Dual or multimodal imaging, i.e. combining two or more imaging modalities
- G16H 30/40: for processing medical images, e.g. editing
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G16H: HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA

## Abstract

Systems and methods for generating a synthetic image are provided. An input medical image in a first modality is received. A synthetic image in a second modality is generated from the input medical image. The synthetic image is upsampled to increase a resolution of the synthetic image. An output image is generated to simulate image processing of the upsampled synthetic image. The output image is output.

## Claims

### Claim 1

A computer implemented method comprising:
receiving an input medical image in a first modality;
generating a synthetic image in a second modality from the input medical image;
upsampling the synthetic image to increase a resolution of the synthetic image;
generating an output image to simulate image processing of the upsampled synthetic image; and
outputting the output image,
wherein:
the input medical image comprises annotations,
generating a synthetic image in a second modality from the input medical image comprises translating the annotations to correspond to the synthetic image, and
upsampling the synthetic image to increase a resolution of the synthetic image comprises resampling the translated annotations to the resolution of the upsampled synthetic image.

### Claim 2 (depends on 1)

The computer implemented method of claim 1, wherein upsampling the synthetic image to increase a resolution of the synthetic image comprises:
performing a first upsampling of the synthetic image to increase a one-dimensional resolution of the synthetic image in a head-feet dimension using a first trained machine learning based model; and
performing a second upsampling of the synthetic image to increase a two-dimensional resolution of the synthetic image using a second trained machine learning based model.

### Claim 3 (depends on 1)

The computer implemented method of claim 1, wherein generating an output image to simulate image processing of the upsampled synthetic image comprises:
generating the output image using a trained machine learning based model.

### Claim 4 (depends on 3)

The computer implemented method of claim 3, wherein the trained machine learning based model is trained using unpaired training images.

### Claim 5 (depends on 1)

The computer implemented method of claim 1, wherein generating a synthetic image in a second modality from the input medical image comprises:
segmenting an anatomical object of interest from the input medical image;
masking regions of the input medical image outside of the segmented anatomical object of interest; and
generating the synthetic image from the masked input medical image.

### Claim 6 (depends on 1)

The computer implemented method of claim 1, wherein the first modality is CT (computed tomography) and the second modality is x-ray, and wherein generating a synthetic image in a second modality from the input medical image comprises:
generating a DRR (digitally reconstructed radiograph) from the input medical image.

### Claim 7 (depends on 1)

The computer implemented method of claim 1, further comprising:
training a machine learning based model for performing a medical imaging analysis task based on the output image.

### Claim 8

A computer implemented method comprising:
receiving an input medical image in a first modality;
generating a synthetic image in a second modality from the input medical image;
upsampling the synthetic image to increase a resolution of the synthetic image;
generating an output image to simulate image processing of the upsampled synthetic image;
outputting the output image;
generating one or more additional synthetic images in the second modality from the input medical image; and
repeating the upsampling, the generating the output image, and the outputting steps for each respective additional synthetic image of the one or more additional synthetic images using the respective additional synthetic image as the synthetic image to generate a plurality of output images.

### Claim 9 (depends on 8)

The computer implemented method of claim 8, wherein upsampling the synthetic image to increase a resolution of the synthetic image comprises:
performing a first upsampling of the synthetic image to increase a one-dimensional resolution of the synthetic image in a head-feet dimension using a first trained machine learning based model; and
performing a second upsampling of the synthetic image to increase a two-dimensional resolution of the synthetic image using a second trained machine learning based model.

### Claim 10

An apparatus comprising:
means for receiving an input medical image in a first modality;
means for generating a synthetic image in a second modality from the input medical image;
means for upsampling the synthetic image to increase a resolution of the synthetic image;
means for generating an output image to simulate image processing of the upsampled synthetic image; and
means for outputting the output image,
wherein:
the input medical image comprises annotations,
the means for generating a synthetic image in a second modality from the input medical image comprises means for translating the annotations to correspond to the synthetic image, and
the means for upsampling the synthetic image to increase a resolution of the synthetic image comprises means for resampling the translated annotations to the resolution of the upsampled synthetic image.

### Claim 11 (depends on 10)

The apparatus of claim 10, wherein the means for upsampling the synthetic image to increase a resolution of the synthetic image comprises:
means for performing a first upsampling of the synthetic image to increase a one-dimensional resolution of the synthetic image in a head-feet dimension using a first trained machine learning based model; and
means for performing a second upsampling of the synthetic image to increase a two-dimensional resolution of the synthetic image using a second trained machine learning based model.

### Claim 12 (depends on 10)

The apparatus of claim 10, wherein the means for generating an output image to simulate image processing of the upsampled synthetic image comprises:
means for generating the output image using a trained machine learning based model.

### Claim 13 (depends on 12)

The apparatus of claim 12, wherein the trained machine learning based model is trained using unpaired training images.

### Claim 14 (depends on 10)

The apparatus of claim 10, wherein the means for generating a synthetic image in a second modality from the input medical image comprises:
means for segmenting an anatomical object of interest from the input medical image;
means for masking regions of the input medical image outside of the segmented anatomical object of interest; and
means for generating the synthetic image from the masked input medical image.

### Claim 15

A non-transitory computer readable medium storing computer program instructions, the computer program instructions when executed by a processor cause the processor to perform operations comprising:
receiving an input medical image in a first modality;
generating a synthetic image in a second modality from the input medical image;
upsampling the synthetic image to increase a resolution of the synthetic image;
generating an output image to simulate image processing of the upsampled synthetic image; and
outputting the output image,
wherein:
the input medical image comprises annotations,
generating a synthetic image in a second modality from the input medical image comprises translating the annotations to correspond to the synthetic image, and
upsampling the synthetic image to increase a resolution of the synthetic image comprises resampling the translated annotations to the resolution of the upsampled synthetic image.

### Claim 16 (depends on 15)

The non-transitory computer readable medium of claim 15, wherein upsampling the synthetic image to increase a resolution of the synthetic image comprises:
performing a first upsampling of the synthetic image to increase a one-dimensional resolution of the synthetic image in a head-feet dimension using a first trained machine learning based model; and
performing a second upsampling of the synthetic image to increase a two-dimensional resolution of the synthetic image using a second trained machine learning based model.

### Claim 17 (depends on 15)

The non-transitory computer readable medium of claim 15, wherein the first modality is CT (computed tomography) and the second modality is x-ray, and wherein generating a synthetic image in a second modality from the input medical image comprises:
generating a DRR (digitally reconstructed radiograph) from the input medical image.

### Claim 18 (depends on 15)

The non-transitory computer readable medium of claim 15, the operations further comprising:
training a machine learning based model for performing a medical imaging analysis task based on the output image.

### Claim 19

A non-transitory computer readable medium storing computer program instructions, the computer program instructions when executed by a processor cause the processor to perform operations comprising:
receiving an input medical image in a first modality;
generating a synthetic image in a second modality from the input medical image;
upsampling the synthetic image to increase a resolution of the synthetic image;
generating an output image to simulate image processing of the upsampled synthetic image;
outputting the output image;
generating one or more additional synthetic images in the second modality from the input medical image; and
repeating the upsampling, the generating the output image, and the outputting operations for each respective additional synthetic image of the one or more additional synthetic images using the respective additional synthetic image as the synthetic image to generate a plurality of output images.

### Claim 20

An apparatus comprising:
means for receiving an input medical image in a first modality;
means for generating a synthetic image in a second modality from the input medical image;
means for upsampling the synthetic image to increase a resolution of the synthetic image;
means for generating an output image to simulate image processing of the upsampled synthetic image;
means for outputting the output image;
means for generating one or more additional synthetic images in the second modality from the input medical image; and
means for repeating the means for upsampling, the means for generating the output image, and the means for outputting for each respective additional synthetic image of the one or more additional synthetic images using the respective additional synthetic image as the synthetic image to generate a plurality of output images.