# US 11,077,320 B1 · Adversarial prediction of radiotherapy treatment plans

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,077,320 B1
- Document ID: US11077320B1
- Status: Granted
- Kind: B1
- Application number: 16/784,919
- Filed: 2020-02-07
- Published: 2021-08-12
- Granted: 2021-08-03
- Earliest claimed: 2020-02-07
- Assignees: Elekta Limited
- Applicants: Elekta, Inc.
- Inventors: Lyndon Stanley Hibbard
- Web page: https://patentgrep.com/patent/US11077320B1

## CPC

- A61N 5/103: {Treatment planning systems}
- A61N 5/1047: {with movement of the radiation head during application of radiation, e.g. for intensity modulated arc therapy or IMAT}
- A61N 5/1031: {using a specific method of dose optimization}
- A61N 5/1039: {using functional images, e.g. PET or MRI}
- A61N 5/1067: {in real time, i.e. during treatment}
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/0475: Generative networks
- G06N 3/08: Learning methods
- G06N 3/09: Supervised learning
- G06N 3/094: Adversarial learning
- G06T 7/0012: {Biomedical image inspection}
- G16H 20/40: relating to mechanical, radiation or invasive therapies, e.g. surgery, laser therapy, dialysis or acupuncture
- G06T 2200/08: involving all processing steps from image acquisition to 3D model generation
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- A61N: ELECTROTHERAPY; MAGNETOTHERAPY; RADIATION THERAPY; ULTRASOUND THERAPY (measurement of bioelectric currents A61B; surgical instruments, devices or methods for transferring non-mechanical forms of energy to or from the body A61B18/00; anaesthetic apparatus in general A61M; incandescent lamps H01K; infrared radiators for heating H05B)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G16H: HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA

## Abstract

Systems and methods are disclosed for generating radiotherapy treatment machine parameters based on projection images of a target anatomy. The systems and methods include operations including receiving a set of pairs of image data for each gantry angle of a radiotherapy treatment machine, wherein each pair of the set of pairs comprises a given projection image that represents a view of an anatomy of a subject from a given gantry angle and a given graphical aperture image of multi-leaf collimator (MLC) leaf positions at the given gantry angle based on the given projection image; training a generative adversarial network (GAN) model based on the set of pairs of image data for each gantry angle; and using the trained GAN model to predict an aperture image of MLC leaf positions for a desired gantry angle based on a projection image that represents a view of an anatomical region of interest.

## Claims

### Claim 1

A computer-implemented method for generating one or more radiotherapy treatment plans, the method comprising:
receiving a set of pairs of image data for each gantry angle of a radiotherapy treatment machine, wherein each pair of the set of pairs comprises a given projection image that represents a view of an anatomy of a subject from a given gantry angle and a given graphical aperture image of multi-leaf collimator (MLC) leaf positions at the given gantry angle based on the given projection image;
training a generative adversarial network (GAN) model based on the set of pairs of image data for each gantry angle; and
using the trained GAN model to predict an aperture image of MLC leaf positions for a desired gantry angle based on a projection image that represents a view of an anatomical region of interest.

### Claim 2 (depends on 1)

The method of claim 1, wherein the GAN comprises a conditional adversarial network (cGAN).

### Claim 3 (depends on 1)

The method of claim 1, wherein the GAN comprises a cycle-consistent generative adversarial network (CycleGAN).

### Claim 4 (depends on 1)

The method of claim 1, wherein the given gantry angle of a first pair of the set of pairs differs from the given gantry angle of a second pair of the set of pairs by a predetermined amount.

### Claim 5 (depends on 1)

The method of claim 1, wherein:
the GAN is configured to train the generative model using a discriminative model;
values applied by the generative model and the discriminative model are established using adversarial training between the discriminative model and the generative model; and
the generative model and the discriminative model comprise respective convolutional neural networks.

### Claim 6 (depends on 5)

The method of claim 5, wherein:
the adversarial training comprises:
training the generative model to generate a first synthetic graphical aperture image representation of the MLC leaf positions at a first gantry angle from a projection image that represents a view of a training subject anatomy from the first gantry angle; and
training the discriminative model to classify the first synthetic graphical aperture image as a synthetic or a real training example graphical aperture image; and
an output of the generative model is used for training the discriminative model and an output of the discriminative model is used for training the generative model.

### Claim 7 (depends on 6)

The method of claim 6, wherein the GAN is trained using a cycle-consistent generative adversarial network (CycleGAN) comprising the generative model and the discriminative model, wherein the generative model is a first generative model and the discriminative model is a first discriminative model, wherein the CycleGAN further comprises:
a second generative model trained to:
process, from a given pair of the set of pairs, a given graphical aperture image representation of the MLC leaf positions at a given gantry angle as an input; and
provide a synthetic projection image that represents a view of a training subject anatomy from the given gantry angle as an output; and
a second discriminative model trained to classify the synthetic projection image as a synthetic or a real projection image.

### Claim 8 (depends on 7)

The method of claim 7, wherein the CycleGAN comprises a first portion to train the first generative model, the first portion being trained to:
obtain a set of training projection images representing different views of a patient anatomy from prior treatments that are paired with training graphical aperture images corresponding to each of the different views, each of the training graphical aperture images being aligned with a respective one of the training projection images;
transmit the set of training projection images to an input of the first generative model to output a first set of graphical aperture images;
receive the first set of graphical aperture images at an input of the first discriminative model to classify the first set of graphical aperture images as a synthetic or real training set of graphical aperture images; and
receive the first set of graphical aperture images at an input of the second generative model to generate a first set of cycle projection images, for calculating cycle-consistency losses.

### Claim 9 (depends on 8)

The method of claim 8, wherein the CycleGAN comprises a second portion that is trained to:
transmit the training graphical aperture images corresponding to each of the different views to the input of the second generative model to output a first set of synthetic projection images;
receive the first set of synthetic projection images at an input of the second discriminative model to classify the first set of synthetic projection images as synthetic or real training projection images; and
receive the first set of synthetic projection images at the input of the first generative model to generate a first set of cycle graphical aperture images for calculating cycle-consistency losses.

### Claim 10 (depends on 9)

The method of claim 9, wherein:
the cycle-consistency losses are generated based on a comparison of the first set of cycle projection images with the set of training projection images and a comparison of the first set of cycle graphical aperture images with the training graphical aperture images;
the first generative model is trained to minimize or reduce a first loss term that represents an expectation of difference between a plurality of synthetic graphical aperture images and respectively paired training graphical aperture images; and
the second generative model is trained to minimize or reduce a second loss term that represents an expectation of difference between a plurality of synthetic projection images and respectively paired training projection images.

### Claim 11 (depends on 1)

The method of claim 1, further comprising:
obtaining radiotherapy treatment machine parameter information representing MLC leaf positions at gantry angles corresponding to each of the views represented by the projection images of the set of pairs of image data and radiotherapy beam intensities corresponding to each of the projection images;
generating training graphical aperture image representations based on the obtained radiotherapy treatment machine parameter information; and
aligning each of the generated training graphical aperture image representations with the corresponding projection images.

### Claim 12 (depends on 11)

The method of claim 11, wherein the training graphical aperture image representations and corresponding projection images are two-dimensional images or three-dimensional images comprising stacks of two-dimensional projection image and graphical aperture image pairs corresponding to an entire treatment fraction.

### Claim 13 (depends on 1)

The method of claim 1, further comprising generating one or more radiotherapy treatment machine parameters from the aperture image predicted by the trained GAN model.

### Claim 14 (depends on 13)

The method of claim 13, further comprising computing a dose volume histogram or a three-dimensional dose distribution based on the generated one or more radiotherapy treatment machine parameters.

### Claim 15 (depends on 13)

The method of claim 13, wherein the one or more parameters include at least one of a gantry angle, an MLC jaw position, an MLC leaf position, or a radiation therapy beam intensity.

### Claim 16 (depends on 13)

The method of claim 13, further comprising computing dosimetric parameters based on the generated one or more radiotherapy treatment machine parameters.

### Claim 17 (depends on 1)

The method of claim 1, wherein the given projection images in the set of pairs of image data is generated by ray tracing or Fourier reconstruction.

### Claim 18

A system for generating one or more radiotherapy treatment plans, the system comprising:
one or more processors configured to perform operations comprising:
receiving a set of pairs of image data for each gantry angle of a radiotherapy treatment machine, wherein each pair of the set of pairs comprises a given projection image that represents a view of an anatomy of a subject from a given gantry angle and a given graphical aperture image of multi-leaf collimator (MLC) leaf positions at the given gantry angle based on the given projection image;
training a generative adversarial network (GAN) model based on the set of pairs of image data for each gantry angle; and
using the trained GAN model to predict an aperture image of MLC leaf positions for a desired gantry angle based on a projection image that represents a view of an anatomical region of interest.

### Claim 19 (depends on 18)

The system of claim 18, wherein the GAN comprises a conditional adversarial network (cGAN) or a cycle-consistent generative adversarial network (CycleGAN).

### Claim 20 (depends on 18)

The system of claim 18, further comprising operations for generating, as the projection images in the set of pairs of image data, a plurality of two-dimensional projection images that represent a plurality of two-dimensional views of the anatomy from a plurality of different gantry angles.

### Claim 21 (depends on 20)

The system of claim 20, wherein the plurality of two-dimensional projection images represents views of the anatomy at a plurality of angles spanning the arc of motion of a gantry of the radiotherapy treatment machine during therapy.

### Claim 22

A non-transitory computer-readable medium comprising non-transitory computer-readable instructions for performing operations comprising:
receiving a set of pairs of image data for each gantry angle of a radiotherapy treatment machine, wherein each pair of the set of pairs comprises a given projection image that represents a view of an anatomy of a subject from a given gantry angle and a given graphical aperture image of multi-leaf collimator (MLC) leaf positions at the given gantry angle based on the given projection image;
training a generative adversarial network (GAN) model based on the set of pairs of image data for each gantry angle; and
using the trained GAN model to predict an aperture image of MLC leaf positions for a desired gantry angle based on a projection image that represents a view of an anatomical region of interest.