# US 12,002,202 B2 · Meta-learning for cardiac MRI segmentation

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,002,202 B2
- Document ID: US12002202B2
- Status: Granted
- Kind: B2
- Application number: 17/397,334
- Filed: 2021-08-09
- Published: 2023-02-09
- Granted: 2024-06-04
- Earliest claimed: 2021-08-09
- Assignees: Merck Sharp & Dohme Corp.
- Applicants: Merck Sharp & Dohme LLC
- Inventors: Dani Kiyasseh; Antong Chen; Albert Joseph Swiston, Jr.; Ronghua Chen
- Web page: https://patentgrep.com/patent/US12002202B2

## CPC

- G06T 7/0012: {Biomedical image inspection}
- G06N 3/045: Combinations of networks
- G06N 3/0455: Auto-encoder networks; Encoder-decoder networks
- G06N 3/08: Learning methods
- G06N 3/0895: Weakly supervised learning, e.g. semi-supervised or self-supervised learning
- G06N 3/0985: Hyperparameter optimisation; Meta-learning; Learning-to-learn
- G06T 7/11: Region-based segmentation
- G06V 10/26: Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion
- G06V 10/82: using neural networks
- G06T 2207/20076: Probabilistic image processing
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/30048: Heart; Cardiac
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

Methods and systems are described for image segmentation. A machine learning model is applied to a set of images to generate results. The results may be obtained as a probability map for each image in the set of images. The model may be trained by accessing a set of labeled images, each image associated with a label indicating a location of a feature within a respective image. An initial set of parameters is accessed. An encoder is initialized with the initial set of parameters. The encoder is applied to the set of labeled images to generate a prediction of a feature location within each image. The initial set of parameters are updated based on the predictions and the label associated with the labeled images. The updated set of parameters and an additional set of parameters generated using a set of unlabeled images are aggregated.

## Claims

### Claim 1

A computer implemented method of segmenting a set of images comprising:
applying a machine learning model to the set of images to generate results, wherein the machine learning model was trained by:
initializing a first encoder with a first set of parameters,
applying the first encoder to a set of unlabeled images to generate a set of representations of the unlabeled images,
updating the first set of parameters based on the set of representations, initializing a second encoder with a second set of parameters,
applying the second encoder to a set of labeled images to generate a set of predictions of a location of a feature within each image in the set of labeled images, wherein each respective label is associated with the location of the feature in the respective labeled image,
updating the second set of parameters based on the set of predictions, aggregating the updated first set of parameters and the updated second set of parameters into a set of aggregated parameters, and
updating the first set of parameters based on the set of aggregated parameters; and
obtaining the results of the machine learning model as a probability map for each image in the set of images.

### Claim 2 (depends on 1)

The method of claim 1, wherein the second set of parameters is the updated first set of parameters.

### Claim 3 (depends on 1)

The method of claim 1, wherein the first encoder is optimized according to a self-supervised loss function.

### Claim 4 (depends on 3)

The method of claim 3, wherein optimizing the first encoder comprises:
augmenting an instance of an image in the set of unlabeled imaged with one or more transformation operators to generate a plurality of augmented instances;
encoding the plurality of augmented instances to generate representations of the plurality of augmented instances; and
applying a similarity metric to the representations of the plurality of augmented instances.

### Claim 5 (depends on 1)

The method of claim 1, wherein the second encoder is optimized according to a cross entropy loss function.

### Claim 6 (depends on 5)

The method of claim 5, wherein optimizing the second encoder comprises:
mapping the set of labeled images into D-dimensional representations with the second encoder; and
mapping the representations into segmentation maps consisting of a set of classes with a decoder.

### Claim 7 (depends on 1)

The method of claim 1, wherein the first encoder is associated with a first task and wherein the second encoder is associated with a second task.

### Claim 8 (depends on 7)

The method of claim 7, wherein aggregating the updated first set of parameters and the updated second set of parameters comprises generating new parameters based on an update step that is defined by:
θ
=
θ
-
α
⁢
∑
t
=
1
T
⁢
(
θ
-
θ
t
)
,
wherein θ are initialization parameters, α is a learning rate, t is a task, and θt are task-specific parameters.

### Claim 9 (depends on 7)

The method of claim 7, wherein aggregating the updated first set of parameters and the updated second set of parameters comprises generating new parameters based on an update step that is defined by:
θ=θ−α∇ET[d(θ,M)], wherein θ are initialization parameters, α is a learning rate, and M is a manifold associated with each task, t, in a set of T tasks.

### Claim 10 (depends on 1)

The method of claim 1, wherein the set of unlabeled images and the set of labeled images are each associated with at least one of a first species, a first chamber, or a first modality, the method further comprising testing the machine learning model with a set of testing images, wherein the set of testing images are associated with at least one of a second species, a second chamber, or a second modality.

### Claim 11

A non-transitory computer-readable storage medium having instructions encoded thereon that, when executed by one or more processors, cause the one or more processors to:
apply a machine learning model to a set of images to generate results, wherein the machine learning model was trained by:
initializing a first encoder with a first set of parameters,
applying the first encoder to a set of unlabeled images to generate a set of representations of the unlabeled images,
updating the first set of parameters based on the set of representations,
initializing a second encoder with a second set of parameters,
applying the second encoder to a set of labeled images to generate a set of predictions of a location of a feature within each image in the set of labeled images, wherein each respective label is associated with the location of the feature in the respective labeled image,
updating the second set of parameters based on the set of predictions,
aggregating the updated first set of parameters and the updated second set of parameters into a set of aggregated parameters, and
updating the first set of parameters based on the set of aggregated parameters; and
obtain the results of the machine learning model as a probability map for each image in the set of images.

### Claim 12 (depends on 11)

The non-transitory computer-readable storage medium of claim 11, wherein the second set of parameters is the updated first set of parameters.

### Claim 13 (depends on 11)

The non-transitory computer-readable storage medium of claim 11, wherein the first encoder is optimized according to a self-supervised loss function.

### Claim 14 (depends on 13)

The non-transitory computer-readable storage medium of claim 13, wherein optimizing the first encoder comprises:
augmenting an instance of an image in the set of unlabeled imaged with one or more transformation operators to generate a plurality of augmented instances;
encoding the plurality of augmented instances to generate representations of the plurality of augmented instances; and
applying a similarity metric to the representations of the plurality of augmented instances.

### Claim 15 (depends on 11)

The non-transitory computer-readable storage medium of claim 11, wherein the second encoder is optimized according to a cross entropy loss function.

### Claim 16 (depends on 15)

The non-transitory computer-readable storage medium of claim 15, wherein optimizing the second encoder comprises:
mapping the set of labeled images into D-dimensional representations with the second encoder; and
mapping the representations into segmentation maps consisting of a set of classes with a decoder.

### Claim 17 (depends on 11)

The non-transitory computer-readable storage medium of claim 11, wherein the first encoder is associated with a first task and wherein the second encoder is associated with a second task.

### Claim 18 (depends on 17)

The non-transitory computer-readable storage medium of claim 17, wherein aggregating the updated first set of parameters and the updated second set of parameters comprises generating new parameters based on an update step that is defined by:
θ
=
θ
-
α
⁢
∑
t
=
1
T
⁢
(
θ
-
θ
t
)
,
wherein θ are initialization parameters, α is a learning rate, t is a task, and θt are task-specific parameters.

### Claim 19 (depends on 17)

The non-transitory computer-readable storage medium of claim 17, wherein aggregating the updated first set of parameters and the updated second set of parameters comprises generating new parameters based on an update step that is defined by:
θ=θ−α∇T[d(θ,M)], wherein θ are initialization parameters, α is a learning rate, and M is a manifold associated with each task, t, in a set of T tasks.

### Claim 20 (depends on 11)

The non-transitory computer-readable storage medium of claim 11, wherein the set of unlabeled images and the set of labeled images are each associated with at least one of a first species, a first chamber, or a first modality, and
wherein the instructions further cause the one or more processors to test the machine learning model with a set of testing images, wherein the set of testing images are associated with at least one of a second species, a second chamber, or a second modality.