US 2025/0028996 A1 · AN ADAPTIVE PERSONALIZED FEDERATED LEARNING METHOD SUPPORTING HETEROGENEOUS MODEL

Abstract

The present invention discloses an adaptive personalized federated learning method supporting heterogeneous model, based on the use of models with different structures by various participants supporting federated learning, learning the dynamic weight used for model ensemble and introducing optimization objectives for model integration in the process of training model parameters, realizing highly accurate personalized federated learning with heterogeneous and self adaptive data, the participants are enabled to benefit from federated learning in scenes with heterogeneous data at different levels. The adaptive personalized federated learning method of the present invention does not need to introduce new hyper parameters, and can be conveniently deployed in the existing federated learning system; comparing with the traditional personalized federated learning method, the present invention has stronger adaptability.

An adaptive personalized federated learning method supporting heterogeneous model, comprising the following steps:

(1) initializing parameters of a global shared model by a central server;
(2) the central server sending the parameters of the global shared model to each participant of the federated learning, after receiving the parameters of the global shared model, the participants updating their own global shared model with the parameters;
(3) the participants performing learning for adaptability to update the weights of private models;
(4) the participants using newly obtained private training data to train both the private models and the globally shared model by using a stochastic gradient descent algorithm;
(5) the participants uploading the parameters of the globally shared model to the central server after one round of iterative training;
(6) after collecting enough parameters of the global shared model, the central server aggregating these model parameters to obtain new parameters of the global shared model, and then returning to step (2) to distribute the new parameters of the global shared model to each participant, and then circulating until the loss function of all models converges or reaches the maximum number of iterations in federated learning;
wherein, the private models are the models held by each participant of federated learning, and the structure and parameters are not disclosed, the structure of the private models held by each participant are different;
wherein, the specific implementation of step (3) is as follows:
the participants first dividing a small portion (such as 5% of the training data) from the obtained private training data as a validation set, and inferring the private models and the global shared model on the validation set, obtaining the prediction output result ppri of the private models and the prediction output result psha of the global shared models, then the participants updating the weight of the private models through the stochastic gradient descent method, and the update expression is as follows:
λ
i
′
=
λ
i
-
η
⁢
∇
λ
i
L
C
⁢
E
(
p
aen
,
y
)
wherein, λi is the weight of the private model before update, λ′i is the weight of the private model after update, n represents learning rate, ∇λi represents LCE (paen y) to λi gradient, LCE (paen, Y) represents cross entropy of paen and y, paen represents the weighted average result of ppri and psha, y is the ground-truth label;
wherein, the loss function expression used for private model training in step (4) is as follows:
L
p
⁢
r
⁢
i
=
L
C
⁢
E
(
P
pri
,
y
)
+
D
K
⁢
L
(
p
p
⁢
r
⁢
i
⁢

p
s
⁢
h
⁢
a
)
+
L
C
⁢
E
(
p
aen
,
y
)
wherein, Lpri is the loss function of the private model, LCE(ppri, y) represents cross entropy of ppri and y, LCE (paen, y) represents cross entropy of paen and y, DKL (ppri∥psha) represents the KL divergence of ppri relative to psha, paen represents the weighted average result of ppri and psha, y is the ground-truth label, psha is the prediction output result of the global shared model;
wherein, the expression of the loss function used for the global shared model training in step (4) is as follows:
L
s
⁢
h
⁢
a
=
L
C
⁢
E
(
p
s
⁢
h
⁢
a
,
y
)
+
D
K
⁢
L
(
p
s
⁢
h
⁢
a
||
p
p
⁢
r
⁢
i
)
+
L
C
⁢
E
(
p
aen
,
y
)
wherein, Lsha is the loss function of the global shared model, LCE(psha,y) represents cross entropy of psha and y, LCE(paen, y) represents cross entropy of paen and y, DKL (psha∥ppri) represents the KL divergence of psha relative to ppri, paen represents the weighted average result of ppri and psha, y is the ground-truth label, ppri is the prediction output result of the private model, psha is the prediction output result of the global shared model.

134 more elements

An adaptive personalized federated learning method supporting heterogeneous model

Life of this patent

An adaptive personalized federated learning method supporting heterogeneous model

Life of this patent

Abstract

Claims6