# US 2019/0318474 A1 · Image synthesis using adversarial networks such as for radiation therapy > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2019/0318474 A1 - Document ID: US20190318474A1 - Status: Published application - Kind: A1 - Application number: 15/952,686 - Filed: 2018-04-13 - Published: 2019-10-17 - Earliest claimed: 2018-04-13 - Assignees: Elekta Limited - Applicants: Elekta, Inc. - Inventors: Xiao Han - Web page: https://patentgrep.com/patent/US20190318474A1 ## CPC - G06V 10/764: using classification, e.g. of video objects - A61N 5/103: {Treatment planning systems} - G06T 7/0012: {Biomedical image inspection} - G06V 10/82: using neural networks - G06T 2207/10081: Computed x-ray tomography [CT] - G06T 2207/10088: Magnetic resonance imaging [MRI] - G06T 2207/20081: Training; Learning - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - A61N: ELECTROTHERAPY; MAGNETOTHERAPY; RADIATION THERAPY; ULTRASOUND THERAPY (measurement of bioelectric currents A61B; surgical instruments, devices or methods for transferring non-mechanical forms of energy to or from the body A61B18/00; anaesthetic apparatus in general A61M; incandescent lamps H01K; infrared radiators for heating H05B) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract A statistical learning technique that does not rely upon paired imaging information is described herein. The technique may be computer-implemented and may be used in order to train a statistical learning model to perform image synthesis, such as in support of radiation therapy treatment planning. In an example, a trained statistical learning model may include a convolutional neural network established as a generator convolutional network, and the generator may be trained at least in part using a separate convolutional neural network established as a discriminator convolutional network. The generator convolutional network and the discriminator convolutional network may form an adversarial network architecture for use during training. After training, the generator convolutional network may be provided for use in synthesis of images, such as to receive imaging data corresponding to a first imaging modality type, and to synthesize imaging data corresponding to a different, second imaging modality type. ## Claims ### Claim 1 A computer-implemented method for synthesizing a medical image using a trained statistical learning model, the method comprising: receiving medical imaging data obtained using a first imaging modality type; applying the trained statistical learning model to the received medical imaging data to synthesize a medical image corresponding to a different second imaging modality type; and providing the synthesized medical image for presentation or for use in further processing; wherein the trained statistical learning model is established at least in part using a similarity determination between training imaging data provided at the model input, the training imaging data corresponding to the first imaging modality type and synthesized imaging data at the model output corresponding to the second imaging modality type; and wherein the trained statistical learning model is established at least in part using a separate statistical learning model, the separate statistical learning model established to discriminate between actual imaging data corresponding to the second imaging modality and the synthesized imaging data. ### Claim 2 (depends on 1) The computer-implemented method of claim 1, wherein the trained statistical learning model comprises a first generator convolutional network and the separate statistical learning model comprises a first discriminator convolutional network. ### Claim 3 (depends on 2) The computer-implemented method of claim 2, wherein the trained statistical learning model is established without requiring use of a second generator convolutional network and a second discriminator convolutional network, the second generator convolutional network trained to synthesize a medical image corresponding to the first imaging modality type from an input image corresponding to the second imaging modality type, and the second discriminator network established to discriminate between synthesized imaging data corresponding to the first imaging modality type and actual imaging data corresponding to the first imaging modality type. ### Claim 4 (depends on 2) The computer-implemented method of claim 2, wherein the first generator convolutional network and the first discriminator convolutional network form an adversarial network architecture, the adversarial network architecture structured to: adjust the first discriminator convolutional network to enhance or maximize a value of an objective over a distribution of inputs to the first discriminator convolutional network; and adjust the first generator convolutional network to reduce or minimize the value of the objective over a distribution of inputs to the first generator convolutional network. ### Claim 5 (depends on 4) The computer-implemented method of claim 4, wherein the objective is represented as a sum of contributions from an adversarial objective and the similarity determination. ### Claim 6 (depends on 4) The computer-implemented method of claim 4, wherein the objective comprises terms representing (1) an expectation value that an input to the first discriminator including actual imaging data is identified as such by the first discriminator and (2) an expectation value that an input to the discriminator including a synthesized image from the first generator is not identified as actual imaging data by the first discriminator. ### Claim 7 (depends on 1) The computer-implemented method of claim 1, wherein the similarity determination comprises at least one of: a mutual information determination; or a cross-correlation determination. ### Claim 8 (depends on 1) The computer-implemented method of claim 1, wherein the first imaging modality type comprises magnetic resonance (MR) imaging; and wherein the second imaging modality type comprises computed tomography (CT) imaging. ### Claim 9 (depends on 1) The computer-implemented method of claim 1, wherein presenting the synthesized image includes displaying the image on an MR-LINAC workstation for use in radiation therapy treatment planning or administration. ### Claim 10 (depends on 1) The computer-implemented method of claim 1, comprising updating, or receiving an update, of a radiation therapy treatment protocol based at least in part upon the synthesized medical image. ### Claim 11 A computer-implemented method for establishing a trained statistical learning model for synthesizing a medical image without requiring paired imaging for training, the method comprising: receiving training medical imaging data corresponding to a first imaging modality type; applying a statistical learning model to the received medical imaging data to synthesize imaging data corresponding to a different second imaging modality type; adjusting the statistical learning model at least in part using a similarity determination between the training imaging data and synthesized imaging data at the model output; and adjusting the statistical learning model at least in part using a separate statistical learning model, the separate statistical learning model established to discriminate between the synthesized imaging data and actual imaging data corresponding to the second imaging modality type. ### Claim 12 (depends on 11) The computer-implemented method of claim 11, wherein the statistical learning model comprises a first generator convolutional network and the separate statistical learning model comprises a first discriminator convolutional network. ### Claim 13 (depends on 12) The computer-implemented method of claim 12, wherein adjusting the statistical learning model includes using a stochastic gradient descent technique. ### Claim 14 (depends on 12) The computer-implemented method of claim 12, wherein the first generator convolutional network and the first discriminator convolutional network form an adversarial network architecture, the adversarial network architecture structured to: adjust the first discriminator convolutional network to enhance or maximize a value of an objective over a distribution of inputs to the first discriminator convolutional network; and adjust the first generator convolutional network to reduce or minimize the value of the objective over a distribution of inputs to the first generator convolutional network. ### Claim 15 (depends on 14) The computer-implemented method of claim 14, wherein the objective is represented as a sum of contributions from an adversarial objective and the similarity determination. ### Claim 16 (depends on 14) The computer-implemented method of claim 14, wherein the objective comprises terms representing (1) an expectation value that an input to the first discriminator including actual imaging data is identified as such by the first discriminator and (2) an expectation value that an input to the discriminator including a synthesized image from the first generator is not identified as actual imaging data by the first discriminator. ### Claim 17 (depends on 11) The computer-implemented method of claim 11, wherein the similarity determination comprises at least one of: a mutual information determination; or a cross-correlation determination. ### Claim 18 (depends on 11) The computer-implemented method of claim 11, wherein the first imaging modality type comprises magnetic resonance (MR) imaging; and wherein the second imaging modality type comprises computed tomography (CT) imaging. ### Claim 19 A system, comprising: processing circuitry comprising at least one processor; and a storage medium comprising instructions, which when executed by the at least one processor, cause the processor to: receive medical imaging data obtained using a first imaging modality type; apply the trained statistical learning model to the received medical imaging data to synthesize a medical image corresponding to a different second imaging modality type; and provide the synthesized medical image for presentation or for use in further processing; wherein the trained statistical learning model is established at least in part using a similarity determination between training imaging data provided at the model input, the training imaging data corresponding to the first imaging modality type and synthesized imaging data at the model output corresponding to the second imaging modality type; and wherein the trained statistical learning model is established at least in part using a separate statistical learning model, the separate statistical learning model established to discriminate between the synthesized imaging data and actual imaging data corresponding to the second imaging modality type. ### Claim 20 (depends on 19) The system of claim 19, wherein the instructions cause the processor to update, or receive an update, of a radiation therapy treatment protocol based at least in part upon the synthesized medical image.