# US 12,602,754 B2 · Dynamic imaging and motion artifact reduction through deep learning

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,602,754 B2
- Document ID: US12602754B2
- Status: Granted
- Kind: B2
- Application number: 17/686,819
- Filed: 2022-03-04
- Published: 2022-09-15
- Granted: 2026-04-14
- Earliest claimed: 2021-03-04
- Assignees: Rensselaer Polytechnic Institute
- Applicants: Rensselaer Polytechnic Institute
- Inventors: Ge Wang; Qing Lyu; Hongming Shan
- Web page: https://patentgrep.com/patent/US12602754B2

## CPC

- G06T 5/73: Deblurring; Sharpening
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU]
- G06N 3/045: Combinations of networks
- G06N 3/0455: Auto-encoder networks; Encoder-decoder networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/0475: Generative networks
- G06N 3/09: Supervised learning
- G06N 3/094: Adversarial learning
- G06N 3/096: Transfer learning
- G06T 3/4046: using neural networks
- G06T 5/60: using machine learning, e.g. neural networks
- G06T 2207/10016: Video; Image sequence
- G06T 2207/10081: Computed x-ray tomography [CT]
- G06T 2207/10088: Magnetic resonance imaging [MRI]
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/20201: Motion blur correction
- G06T 2207/30048: Heart; Cardiac
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

A deep neural network for motion artifact reduction includes a transformer neural network, and a synthesis neural network. The transformer neural network is configured to receive input image data comprising a target image, a prior image, and a subsequent image. The transformer neural network is configured to mitigate a temporal aspect of the motion artifact, and to provide a transformer output related to the input data. The synthesis neural network is configured to receive the transformer output, and to provide at least one output image having a reduced motion artifact relative to the input image data.

## Claims

### Claim 1

A deep neural network for motion artifact reduction in medical imaging data, the deep neural network comprising:
a transformer neural network configured to receive input image data comprising a target image, a prior image, and a subsequent image, the transformer neural network configured to mitigate a temporal aspect of at least one motion artifact in the input image data, and to provide a transformer output related to the input data; and
a synthesis neural network configured to receive the transformer output from the transformer neural network, and to provide at least one output image having a reduced motion artifact relative to the input image data;
wherein the deep neural network was trained using training input data comprising input data comprising input image data that is at least one of low resolution and blurry from least one of dynamic imaging and motion artifacts and using training output data comprising ground truth image data; and
wherein the transformer neural network comprises a temporal stage and a multiscale stage, wherein the temporal stage comprises a forward convolutional long short-term memory (ConvLSTM) network and a backward ConvLSTM network, each forward and backward ConvLSTM network comprising a plurality of ConvLSTM and rectified linear unit (ConvLSTMReLU) blocks in which each rectified linear unit corresponds to an activation function, in which each block produces a block output, and the multiscale stage comprises a plurality of convolutional neural networks (CNN) disposed in parallel with one another, each CNN being configured to receive the input image and coupled to a concatenation block, wherein a resulting CNN output is received by the concatenation block, the multiscale stage further comprising a separate CNN configured to receive a concatenation output from the concatenation block and in which the synthesis neural network comprises an encoder circuitry and a decoder circuitry in which the encoder circuitry is configured to receive a plurality of temporal features from the block outputs of the temporal stage and a multiscale stage output which is the concatenation output from the multiscale stage of the transformer neural network as the transformer output.

### Claim 2 (depends on 1)

The deep neural network of claim 1, wherein the transformer neural network and the synthesis neural network are trained in a Wasserstein generative adversarial network (WGAN) framework.

### Claim 3 (depends on 1)

The deep neural network of claim 1, wherein the input image data is selected from the group comprising magnetic resonance imaging (MRI) image data and computed tomography (CT) image data.

### Claim 4 (depends on 1)

The deep neural network of claim 1, wherein the transformer neural network and the synthesis neural network are configured to extract both temporal features and spatial features.

### Claim 5 (depends on 1)

The deep neural network of claim 1, wherein the transformer neural network or the synthesis neural network is trained using synthetic data.

### Claim 6 (depends on 5)

The deep neural network of claim 5, wherein the training is fine-tuned using actual data.

### Claim 7

A deep learning system for motion artifact reduction in medical imaging data, the deep learning system comprising:
a computing device comprising a processor, a memory, an input/output circuitry, and a data store;
a deep learning module configured to receive input data; and
a deep neural network comprising a transformer neural network configured to receive input image data comprising a target image, a prior image, and a subsequent image, the transformer neural network configured to mitigate a temporal aspect of at least one motion artifact in the input image data, and to provide a transformer output related to the input data, and a synthesis neural network configured to receive the transformer output, and to provide at least one output image having a reduced motion artifact relative to the input image data;
wherein the deep neural network was trained using training input data comprising input image data that is at least one of low resolution and blurry from at least one of dynamic imaging and motion artifacts and using training output data comprising ground truth image data; and
wherein the transformer neural network comprises a temporal stage and a multiscale stage, in which the temporal stage comprises a forward convolutional long short-term memory (ConvLSTM) network and a backward ConvLSTM network, each forward and backward ConvLSTM network comprising a plurality of ConvLSTM and rectified linear unit (ConvLSTMReLU) blocks in which each rectified linear unit corresponds to an activation function and each block produces a block output, and the multiscale stage comprises a plurality of convolutional neural networks disposed in parallel and coupled to a concatenation block, each CNN being configured to receive the input image wherein a resulting CNN output is received by the concatenation block, and a separate CNN that is configured to receive a concatenation output from the concatenation block and in which the synthesis neural network comprises an encoder circuitry and a decoder circuitry, wherein the encoder circuitry is configured to receive a plurality of temporal features from the block outputs of the temporal stage and a multiscale stage output which is the concatenation output from the multiscale stage as the transformer output.

### Claim 8 (depends on 7)

The deep learning system of claim 7, wherein the transformer neural network and the synthesis neural network are trained in a Wasserstein generative adversarial network (WGAN) framework.

### Claim 9 (depends on 7)

The deep learning system of claim 7, wherein the input image data is selected from the group comprising magnetic resonance imaging (MRI) image data and computed tomography (CT) image data.

### Claim 10 (depends on 7)

The deep learning system of claim 7, wherein the transformer neural network and the synthesis neural network are configured to extract both temporal features and spatial features.

### Claim 11 (depends on 7)

The deep learning system of claim 7, wherein the transformer neural network or the synthesis neural network is trained using synthetic data.

### Claim 12 (depends on 11)

The deep learning system of claim 11, wherein the training is fine-tuned using actual data.