# US 2024/0005481 A1 · Apparatus, method, and non-transitory computer-readable storage medium for combining real-number-based and complex-number-based images

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2024/0005481 A1
- Document ID: US20240005481A1
- Status: Published application
- Kind: A1
- Application number: 17/853,456
- Filed: 2022-06-29
- Published: 2024-01-04
- Earliest claimed: 2022-06-29
- Assignees: Canon Medical Systems Corporation
- Applicants: Canon Medical Systems Corporation
- Inventors: Hassan HAJI-VALIZADEH
- Web page: https://patentgrep.com/patent/US20240005481A1

## CPC

- G06T 7/0012: {Biomedical image inspection}
- G01R 33/5608: {Data processing and visualization specially adapted for MR, e.g. for feature analysis and pattern recognition on the basis of measured MR data, segmentation of measured MR data, edge contour detection on the basis of measured MR data, for enhancing measured MR data in terms of signal-to-noise ratio by means of noise filtering or apodization, for enhancing measured MR data in terms of resolution by means for deblurring, windowing, zero filling, or generation of gray-scaled images, colour-coded images or images displaying vectors instead of pixels (image data processing or generation, in general G06T)}
- G01S 7/52028: {using digital techniques}
- G01S 15/8977: {using special techniques for image reconstruction, e.g. FFT, geometrical transformations, spatial deconvolution, time deconvolution}
- G06N 3/045: Combinations of networks
- G06N 3/08: Learning methods
- G06T 5/50: using two or more images, e.g. averaging or subtraction
- G06T 5/60: using machine learning, e.g. neural networks
- G06T 5/70: Denoising; Smoothing
- G01R 33/4804: {Spatially selective measurement of temperature or pH}
- G01R 33/4828: {Resolving the MR signals of different chemical species, e.g. water-fat imaging}
- G06T 2207/10088: Magnetic resonance imaging [MRI]
- G06T 2207/10132: Ultrasound image
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G01R: MEASURING ELECTRIC VARIABLES; MEASURING MAGNETIC VARIABLES (indicating correct tuning of resonant circuits H03J3/12)
- G01S: RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

The present disclosure relates to a real-number-based neural network operating in combination with a complex-number-based neural network to perform image processing (e.g., using phase-based medical images). In one embodiment, a method includes, but is not limited to, applying, to inputs of a first trained neural network trained to process real-number-based images, first image data generated from real-number-based measurements obtained by imaging a subject; applying, to inputs of a second trained neural network trained to process complex-number-based images, second image data generated from complex-number-based measurements obtained by imaging the subject; and combining a first output of the first trained neural network and a second output of the second trained neural network to produce a combined image, based on the first image data and the second image data.

## Claims

### Claim 1

An apparatus for performing image processing, comprising:
processing circuitry configured to
apply, to inputs of a first trained neural network trained to process real-number-based images, first image data generated from real-number-based measurements obtained by imaging a subject;
apply, to inputs of a second trained neural network trained to process complex-number-based images, second image data generated from complex-number-based measurements obtained by imaging the subject; and
combine a first output of the first trained neural network and a second output of the second trained neural network to produce a combined image, based on the first image data and the second image data.

### Claim 2 (depends on 1)

The apparatus according to claim 1, wherein the first and second trained neural networks are deep learning-based neural networks.

### Claim 3 (depends on 1)

The apparatus according to claim 1, wherein the processing circuitry is further configured to produce the combined image by calculating a weighted combination of the first and second outputs.

### Claim 4 (depends on 1)

The apparatus according to claim 1, wherein the processing circuitry is further configured to produce the combined image by applying the first and second outputs to a third neural network trained to produce the combined image from the real-number-based measurements and from the complex-number-based measurements.

### Claim 5 (depends on 1)

The apparatus according to claim 1, wherein the processing circuitry is further configured to apply, to inputs of a third trained neural network trained to process at least one of real-number-based images and complex-number-based images, third image data from measurements obtained by imaging the subject, and
wherein the processing circuitry is further configured to produce the combined image by combining a third output of the third trained neural network with the first and second outputs.

### Claim 6 (depends on 1)

The apparatus according to claim 1, further comprising the first trained neural network and the second trained neural network, wherein the first trained neural network comprises a magnitude-based neural network trained to process magnitude-based images, and the second trained neural network comprises a phase-based neural network trained to process phase-based images.

### Claim 7 (depends on 6)

The apparatus according to claim 6, wherein the phase-based neural network trained to process phase-based images comprises a phase-difference-based neural network trained to process phase-difference-based images.

### Claim 8 (depends on 1)

The apparatus according to claim 1, wherein the first image data has a first resolution, the second image data has a second resolution, and the combined image has a third resolution, and
wherein the third resolution is greater than at least one of the first and second resolutions.

### Claim 9 (depends on 1)

The apparatus according to claim 1, wherein the processing circuitry is further configured to generate a mask image from the first image data, and
wherein the second image data is generated by applying the mask image to third image data generated from the complex-number-based measurements obtained by imaging the subject.

### Claim 10 (depends on 1)

The apparatus according to claim 1, wherein the combined image generated by the processing circuitry has fewer artifacts than at least one of the first image data and the second image data.

### Claim 11 (depends on 1)

The apparatus according to claim 1, wherein the combined image generated by the processing circuitry has less noise than at least one of the first image data and the second image data.

### Claim 12 (depends on 1)

The apparatus according to claim 1, wherein the first image data and the second image data are dual-echo sequence data acquired by a magnetic resonance imaging device.

### Claim 13 (depends on 1)

The apparatus according to claim 1, wherein the first image data and the second image data are data acquired by an ultrasound probe.

### Claim 14

An image processing method, comprising:
applying, to inputs of a first trained neural network trained to process real-number-based images, first image data generated from real-number-based measurements obtained by imaging a subject;
applying, to inputs of a second trained neural network trained to process complex-number-based images, second image data generated from complex-number-based measurements obtained by imaging the subject; and
combining a first output of the first trained neural network and a second output of the second trained neural network to produce a combined image, based on the first image data and the second image data.

### Claim 15

A non-transitory computer-readable storage medium storing computer-readable instructions that, when executed by a computer, cause the computer to perform an image processing method, comprising:
applying, to inputs of a first trained neural network trained to process real-number-based images, first image data generated from real-number-based measurements obtained by imaging a subject;
applying, to inputs of a second trained neural network, trained to process complex-number-based images, second image data generated from complex-number-based measurements obtained by imaging the subject; and
combining a first output of the first trained neural network and a second output of the second trained neural network to produce a combined image, based on the first image data and the second image data.
