# US 2022/0044394 A1 · User-guided domain adaptation for rapid annotation from user interactions for pathological organ segmentation

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2022/0044394 A1
- Document ID: US20220044394A1
- Status: Published application
- Kind: A1
- Application number: 17/138,251
- Filed: 2020-12-30
- Published: 2022-02-10
- Earliest claimed: 2020-08-05
- Assignees: Ping An Technology (Shenzhen) Co., Ltd.
- Applicants: Ping An Technology (Shenzhen) Co., Ltd.
- Inventors: Adam P. HARRISON; Ashwin RAJU
- Web page: https://patentgrep.com/patent/US20220044394A1

## CPC

- G06T 7/0012: {Biomedical image inspection}
- G06T 7/11: Region-based segmentation
- G06T 7/344: {involving models}
- G16H 30/40: for processing medical images, e.g. editing
- G16H 50/20: for computer-aided diagnosis, e.g. based on medical expert systems
- G16H 50/70: for mining of medical data, e.g. analysing previous cases of other patients
- G06T 2207/10081: Computed x-ray tomography [CT]
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/20092: Interactive image processing based on input by user
- G06T 2207/30056: Liver; Hepatic
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G16H: HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA

## Abstract

The present disclosure provides a computer-implemented method, a device, and a computer program product using a user-guided domain adaptation (UGDA) architecture. The method includes training a combined model using a source image dataset by minimizing a supervised loss of the combined model to obtain first sharing weights for a first FCN and second sharing weights for a second FCN; training a discriminator by inputting extreme-point/mask prediction pairs for each of the source image dataset and a target image dataset and by minimizing a discriminator loss to obtain discriminator weights; and finetuning the combined model by predicting extreme-point/mask prediction pairs for the target image dataset to fool the discriminator by matching a distribution of the extreme-point/mask prediction pairs for the target image dataset with a distribution of the extreme-point/mask prediction pairs for the source image dataset.

## Claims

### Claim 1

A computer-implemented user-guided domain adaptation (UGDA) method for medical image annotation, comprising:
using a labelled source image dataset of medical images, training a combined model, including a first fully convolutional network (FCN) and a second FCN chained with the first FCN, by minimizing a supervised loss of the combined model to obtain one or more first sharing weights for the first FCN and one or more second sharing weights for the second FCN;
using the combined model to generate extreme-point predictions, based on the one or more first sharing weights, and generate mask predictions, based on the one or more second sharing weights, for each of the source image dataset and a target image dataset;
training a discriminator by inputting extreme-point/mask prediction pairs for each of the source image dataset and the target image dataset and by minimizing a discriminator loss to obtain one or more discriminator weights; and
finetuning the combined model by predicting extreme-point/mask prediction pairs for the target image dataset to fool the discriminator having the one or more discriminator weights and matching a distribution of the extreme-point/mask prediction pairs for the target image dataset with a distribution of the extreme-point/mask prediction pairs of the source image dataset.

### Claim 2 (depends on 1)

The method according to claim 1, wherein predicting the extreme-point/mask prediction pairs for the target image dataset includes:
obtaining the extreme-point predictions of the target image dataset; and
using the obtained extreme-point predictions as anchors to obtain mask predictions thereof, wherein an adversarial loss is used to guide the mask predictions to properly align with the obtained extreme-point predictions.

### Claim 3 (depends on 2)

The method according to claim 2, wherein the adversarial loss is computed according to:
ℒ
adv
=
1
N
t
⁢
∑
D
t
⁢
ℓ
bce
(
d
(
{
Y
⋓
,
E
⋓
}
,
1
)
)
wherein adv denotes the adversarial loss, bce denotes a cross-entropy loss; Nt denotes cardinality of the target image dataset; Dt denotes a volume of the target image dataset; {hacek over (Y)} and {hacek over (E)} denote the mask predictions and the extreme-point predictions; and 1 denotes a label indicating the extreme-point/mask prediction pairs come from the source image dataset.

### Claim 4 (depends on 1)

The method according to claim 1, wherein:
the first FCN is configured to predict extreme points which are inputted into the second FCN; and
the second FCN is configured to predict masks based on the extreme-point predictions of the first FCN.

### Claim 5 (depends on 1)

The method according to claim 1, wherein:
the supervised loss is calculated based on an extreme-point loss and a segmentation loss.

### Claim 6 (depends on 1)

The method according to claim 1, wherein the discriminator loss is computed according to:
ℒ
d
=
1
N
s
⁢
∑
D
s
⁢
ℓ
bce
(
d
(
{
Y
⋓
,
E
⋓
}
,
1
)
)
+
1
N
t
⁢
ℓ
bce
(
d
(
{
Y
⋓
,
E
⋓
}
,
0
)
)
wherein d denotes the discriminator loss; bce denotes a cross-entropy loss; Ns denotes cardinality of the source image dataset; Nt denotes cardinality of the target image dataset; Ds denotes a volume of the source image dataset; Dt denotes a volume of the target image dataset; {hacek over (Y)} and {hacek over (E)} denote the mask predictions and the extreme-point predictions; and 0 and 1 denote labels indicating the extreme-point/mask prediction pairs come from the target image dataset and the source image dataset, respectively.

### Claim 7 (depends on 3)

The method according to claim 3, wherein:
the overall training for the UGDA method is to minimize a total loss which is computed according to the supervised loss and the adversarial loss.

### Claim 8 (depends on 1)

The method according to claim 1, wherein:
the source image dataset includes volumes for pathological and healthy organs; and the target image dataset includes volumes for pathological organs, with etiologies/characteristics unseen in the source image dataset.

### Claim 9 (depends on 1)

The method according to claim 1, wherein:
the first FCN outputs a plurality of Gaussian heatmaps to the second FCN for predicting one mask, each Gaussian heatmap corresponding to one extreme point.

### Claim 10 (depends on 1)

The method according to claim 1, wherein:
the target image dataset includes both user interaction (UI)-labelled and un-labelled volumes; and the source image dataset includes fully supervised volumes with masks.

### Claim 11

A device for user-guided domain adaptation (UGDA), comprising:
a memory, containing a computer program stored thereon; and
a processor, coupled with the memory and configured, when the computer program being executed, to perform a method including:
using a labelled source image dataset of medical images, training a combined model, including a first fully convolutional network (FCN) and a second FCN chained with the first FCN, by minimizing a supervised loss of the combined model to obtain one or more first sharing weights for the first FCN and one or more second sharing weights for the second FCN;
using the combined model to generate extreme-point predictions, based on the one or more first sharing weights, and generate mask predictions, based on the one or more second sharing weights, for each of the source image dataset and a target image dataset;
training a discriminator by inputting extreme-point/mask prediction pairs for each of the source image dataset and the target image dataset and by minimizing a discriminator loss to obtain one or more discriminator weights; and
finetuning the combined model by predicting extreme-point/mask prediction pairs for the target image dataset to fool the discriminator having the one or more discriminator weights and matching a distribution of the extreme-point/mask prediction pairs for the target image dataset with a distribution of the extreme-point/mask prediction pairs of the source image dataset.

### Claim 12 (depends on 11)

The device according to claim 11, wherein predicting the extreme-point/mask prediction pairs for the target image dataset includes:
obtaining the extreme-point predictions of the target image dataset; and
using the obtained extreme-point predictions as anchors to obtain mask predictions thereof, wherein an adversarial loss is used to guide the mask predictions to properly align with the obtained extreme-point predictions.

### Claim 13 (depends on 11)

The device according to claim 11, wherein:
the first FCN is configured to predict extreme points which are inputted into the second FCN; and
the second FCN is configured to predict masks based on the extreme-point predictions of the first FCN.

### Claim 14 (depends on 11)

The device according to claim 11, wherein:
the first FCN outputs a plurality of Gaussian heatmaps to the second FCN for predicting one mask, each Gaussian heatmap corresponding to one extreme point.

### Claim 15

A computer program product comprising a non-transitory computer-readable storage medium and program instructions stored therein, the program instructions being configured to be executable by a computer to cause the computer to implement operations comprising:
using a labelled source image dataset of medical images, training a combined model, including a first fully convolutional network (FCN) and a second FCN chained with the first FCN, by minimizing a supervised loss of the combined model to obtain one or more first sharing weights for the first FCN and one or more second sharing weights for the second FCN;
using the combined model to generate extreme-point predictions, based on the one or more first sharing weights, and generate mask predictions, based on the one or more second sharing weights, for each of the source image dataset and a target image dataset;
training a discriminator by inputting extreme-point/mask prediction pairs for each of the source image dataset and the target image dataset and by minimizing a discriminator loss to obtain one or more discriminator weights; and
finetuning the combined model by predicting extreme-point/mask prediction pairs for the target image dataset to fool the discriminator having the one or more discriminator weights and matching a distribution of the extreme-point/mask prediction pairs for the target image dataset with a distribution of the extreme-point/mask prediction pairs of the source image dataset.

### Claim 16 (depends on 15)

The product according to claim 15, wherein predicting the extreme-point/mask prediction pairs for the target image dataset includes:
obtaining the extreme-point predictions of the target image dataset; and
using the obtained extreme-point predictions as anchors to obtain mask predictions thereof, wherein an adversarial loss is used to guide the mask predictions to properly align with the obtained extreme-point predictions.

### Claim 17 (depends on 15)

The product according to claim 15, wherein:
the first FCN is configured to predict extreme points which are inputted into the second FCN; and
the second FCN is configured to predict masks based on the extreme-point predictions of the first FCN.

### Claim 18 (depends on 15)

The product according to claim 15, wherein:
the first FCN outputs a plurality of Gaussian heatmaps to the second FCN for predicting one mask, each Gaussian heatmap corresponding to one extreme point.
