US 11,960,568 B2 · Model and method for multi-source domain adaptation by aligning partial features

Abstract

A multi-source domain adaptation model by aligning partial features includes a general feature extraction module, a feature selection module for partial feature extraction with the dedicated loss function, three partial feature alignment losses, and two classifiers for adversarial training, where the three partial feature alignment losses include an intra-class partial feature alignment loss, an inter-domain partial feature alignment loss, and an inter-class partial feature alignment loss. With the partial features extracted by the general feature extraction module and the feature selection module following three different partial feature alignment losses, the model is capable of clustering the samples from the identical categories and isolating samples from distinct classes.

A multi-source domain adaptation model by aligning partial features of different domains, comprising:

a general feature extraction module;
a feature selection module for partial feature extraction and a corresponding objective function thereof;
three partial feature alignment losses, the three partial feature alignment losses comprising an intra-class partial feature alignment loss, an inter-domain partial feature alignment loss, and an inter-class partial feature alignment loss;
and
two classifiers for adversarial training;
wherein:
the general feature extraction module is configured to acquire raw feature maps through conventional convolutional neural networks (CNN) and residual neural networks ResNet-101;
the raw feature maps are an output of the general feature extraction module for simple and complicated images;
L1 distances between the raw feature maps of different pairs of the source domain and target domain are used as inputs of the feature selection module for partial feature extraction;
the convolutional neural networks comprise three convolutional layers and two fully connected layers, and a dimension of a last layer of the networks is 2048;
the residual neural networks utilize ResNet-101, with a last classification layer removed, to obtain feature maps of 2048 output dimensions of the residual neural networks;
the feature selection module for partial feature extraction is built with two fully connected layers and utilizes the L1 distances as an input to analyze highly-related feature dimensions between source domains and target domains;
an output of the feature selection module is a feature selection vector for an original feature map, which is applied to the original feature map to conduct a refined feature map through dot product;
on the refined feature map, a loss function of the feature selection module for partial feature extraction, the intra-class partial feature alignment loss, the inter-domain partial feature alignment loss, and the inter-class partial feature alignment loss are calculated, respectively, and a weighted combination of the loss functions and a classification loss is used as an overall loss function;
the loss function of the feature selection module for partial feature extraction is:
L
p
=
1
N
S
⁢
∑
i
=
1
N
S

𝔼
⁡
(
F
S
i
k
)
-
𝔼
⁡
(
F
T
k
)

2
+
(
N
S
2
)
-
1
⁢
∑
i
=
1
N
S
-
1
∑
j
=
i
+
1
N
S

𝔼
⁡
(
F
S
i
k
)
-
𝔼
⁡
(
F
S
j
k
)

2
+
λ
r
⁢
e
⁢
g
⁢
∑
(
1
-
v
_
)
2
,
where FSi=G(Si)·vi and FT=G(T)·v represent refined features of the source domains and the target domains, respectively, and k is an order of a high-order moment distance;
λreg is a weight parameter for regularization;
NS is a number of domains;
(
N
S
2
)
=
N
S
(
N
S
-
1
)
2
is a combinatorial number, and
(
N
S
2
)
-
1
is it's reciprocal;
vi stands for a feature selection vector for source domain i and v is an average over the feature selection vectors of each source domain;
Si represents a batch of sample from a source domain i, and T represents a batch of sample from the target domain;
(*) represents the calculation of an expected value;
G stands for a general feature extractor;
G(Si) is the raw feature map extracted from the source domain i, and G(T) is the raw feature map extracted from the target domain;
the three partial alignment loss functions are calculated on top of class centroids on the refined feature map, which is defined as follows:
f
i
c
=
1
n
c
⁢
∑
F
S
i
c
,
where fic represents a centroid of class c in the source domain i;
F
S
i
c
stands for refined features of samples that belong to class c from the source domain i, and nc represents a number of samples in a training batch;
a class centroid of the target domain is defined as follows:
f
T
c
=
1
n
c
⁢
∑
F
T
c
,
where fTc represents a centroid of class c in the target domain;
FTc stands for the refined features of samples that belong to class c from the target domain, and nc represents the number of samples in the training batch;
to preserve information gained from previous training batches of the multi-source domain adaptation model by aligning partial features, an exponential moving average is utilized to update all of the centroids during each batch:
f
i
c
b
=
β
c
⁢
f
i
c
b
-
1
+
(
1
-
β
c
)
⁢
1
n
c
⁢
∑
F
S
i
c
b
,
where βc is a weight parameter for previously calculated centroids;
ficb and
F
S
i
c
b
represent a maintained centroid and refined features of class c in the source domain i during batch b, respectively;
the calculations of target centroids fTcb are the same as that of ficb;
the intra-class partial feature alignment loss is:
L
c
=
∑
i
⁢
∑
c
⁢

𝔼
⁡
(
(
F
i
c
b
)
k
)
-
𝔼
⁡
(
(
F
S
i
c
b
)
k
)

+
∑
c
⁢

𝔼
⁡
(
(
F
T
c
b
)
k
)
-
𝔼
⁡
(
(
F
T
c
b
)
k
)

;
where k stands for an order of high-order moment;
the inter-domain partial feature alignment loss is:
L
dom
=
∑
i
∑
c
(
f
T
c
b
-
f
i
c
b
)
2
,
where fTcb and ficb represent the above-mentioned centroids of class c of the target domain and the source domain i, respectively;
the inter-class partial feature alignment loss is:
L
disc
=
∑
i
∑
j
(
B
-

f
Ti
b
-
f
Tj
b

)
2
,
where fTib and fTjb represent the centroids of class i and class j of the target domain, respectively;
B is a bias hyper-parameter set for restricting the degree of discreteness of different class centroids on the feature map of the target domain;
an overall loss function for the multi-source domain adaptation model by aligning partial features is:
L=Ls+λpLp+λcLc+λdomLdom+λdiscLdisc,
where Ls is a classification cross-entropy loss for source domains, which is a sum of the cross-entropy losses of two classifiers during adversarial training;
λp, λc, λdom, λdisc are pre-defined weight parameters for Lp, Lc, Ldom, Ldisc, respectively.

336 more elements

Model and method for multi-source domain adaptation by aligning partial features

Life of this patent

Model and method for multi-source domain adaptation by aligning partial features

Life of this patent

Abstract

Claims2