US 2026/0250168 A1 · Adaptive critic control algorithm integrating knowledge transfer for wastewater treatment process

Abstract

An adaptive critic control algorithm integrating knowledge transfer for a wastewater treatment process is provided. This algorithm is used to achieve precise tracking of the dissolved oxygen concentration to the set value during the wastewater treatment process. When realizing the optimal control of a wastewater treatment system with an unknown model, the direct control strategy in the conventional adaptive critic control (ACC) algorithm is often sensitive to system disturbances, which may lead to a sharp decline in control performance. The present invention takes both knowledge transfer and incremental control into consideration. First, the anti-interference performance is effectively improved by optimizing the control strategy of the conventional ACC algorithm into an incremental form. Second, by leveraging the historical operation data of the system, knowledge transfer is implemented among different control methods. As a result, the initial control performance of the present invention during the early training stage has been effectively improved. Thirdly, a new type of utility function with a soft strategy constraint mechanism is designed, which enhances the stability during the online optimization process. Finally, the effectiveness of the present invention has been verified through experiments.

An adaptive critic control method for a wastewater treatment process that integrates knowledge transfer, comprising following steps:

Step 1:
establish an optimal tracking problem regarding dissolved oxygen (DO) concentration in the wastewater treatment process;
a wastewater treatment system can be represented as a class of nonlinear system as follows:
x
k
+
1
=
H
⁡
(
x
k
,
U
k
)
,
k
=
0
,
1
,
2
,
…
(
1
)
where a system state xk∈ represents DO concentration in a tank (1.8) at k moment, a control input uk∈ represents oxygen transfer coefficient at the k moment, H(⋅,⋅) represents an unknown wastewater treatment system function, and represents a set of all positive real numbers;
a set value dk of the DO concentration is expressed as
d
k
=
δ
⁡
(
k
)
(
2
)
where δ(⋅) is a set value function;
according to engineering experience, the set value of the DO concentration is usually selected as 2 mg/L;
a tracking error e between the DO concentration and its set value is defined as
e
K
=
x
k
-
d
k
(
3
)
design an incremental control strategy η(ek), such that the control input satisfies the following equation:
{
Δ
⁢
u
k
=
η
⁡
(
e
k
)
u
k
=
u
k
-
1
+
Δ
⁢
u
k
(
4
)
a utility function is proposed as follows:
U
⁡
(
e
μ
,
Δ
⁢
u
k
)
=
α
1
⁢
e
k
2
+
α
2
⁢
Δ
⁢
u
k
2
+
α
3
⁢
S
⁡
(
e
k
,
Δ
⁢
u
k
)
(
5
)
where α1, α2, and α3 are constant weights greater than 0;
S(ek,Δuk) is defined as
S
⁡
(
e
k
,
Δ
⁢
u
k
)
=
e
k
2
(
Δ
⁢
u
k
-
ϑ
⁡
(
e
k
)
)
2
(
6
)
with ϑ(e)=−121ek+102ek-1−ek-2;
a value function V(ek,Δuk) is defined as
V
⁡
(
e
k
,
Δ
⁢
u
k
)
=
∑
p
=
k
∞
r
p
-
k
⁢
U
⁡
(
e
p
+
1
,
Δ
⁢
u
p
+
1
)
=
U
⁡
(
e
k
+
1
,
Δ
⁢
u
k
+
1
)
+
γ
⁢
V
⁡
(
e
k
+
1
,
Δ
⁢
u
k
+
1
)
(
7
)
where γ∈(0,1) is a discount factor;
an optimal value function V*(ek,Δuk) and an optimal incremental control strategy η*(ek) are respectively defined as
V
*
(
e
k
,
Δ
⁢
u
k
)
=
min
Δ
⁢
u
k
{
U
⁡
(
e
k
+
1
,
Δ
⁢
u
k
+
1
)
⁢
γ
⁢
V
*
(
e
k
+
1
,
Δ
⁢
u
k
+
1
)
}
(
8
)
and
η
*
(
e
k
)
=
arg
min
Δ
⁢
u
k
{
U
⁡
(
e
k
+
1
,
Δ
⁢
u
k
+
1
)
+
γ
⁢
V
*
(
e
k
+
1
,
Δ
⁢
u
k
+
1
)
}
(
9
)
based on an action-critic online optimization mechanism and function approximation capability of a neural network, wastewater treatment process can continuously approach the optimal value function V*(ek,Δuk) and the optimal incremental control strategy η*(ek);
Step 2:
complete initialization of the control strategy using knowledge transfer technology;
an action network is used to approximate the optimal incremental control strategy η*(ek), and a critic network is used to approximate the optimal value function V*(ek,Δuk);
both the action network and the critic network are back-propagation neural networks composed of an input layer, a hidden layer, and an output layer;
establish a dataset D={(e′j,Δu′j)|j=1, 2, . . . , n} using historical operation data of the wastewater treatment system under the action of an incremental PID control algorithm ϑ(ek), where e′j and Δu′j respectively represent j-th tracking error sample and incremental control input sample, and n is a total number of samples in the dataset D;
a pre-training of the action network is completed based on the dataset D;
during the pre-training process, an output of the action network can be expressed as
η
ˆ
(
e
j
′
)
=
w
a
⁢
2
T
⁢
ϕ
⁡
(
w
a
⁢
1
T
⁢
e
j
′
)
(
10
)
where wa1(k)∈ and wa2(k)∈ are weight vectors of the action network;
ha=10 represents the number of neurons in the hidden layer of the action network;
an activation function φ(⋅) is set as hyperbolic tangent function;
an approximation error of an execution network during the pre-training process can be expressed as
ξ
⁡
(
e
j
′
)
=
η
ˆ
(
e
j
′
)
-
Δ
⁢
u
j
′
(
11
)
a performance index function of the execution network during the pre-training process can be expressed as
Ψ
⁡
(
e
′
)
=
1
n
⁢
∑
j
=
1
n
1
2
⁢
ξ
2
(
e
j
′
)
(
12
)
where e′=[e′1, e′2, . . . , e′n] is an input of the performance index function;
an update rule of the execution network weights during the pre-training process can be expressed as
w
a
⁢
1
:=
w
a
⁢
1
-
β
a
⁢
∂
Ψ
⁡
(
e
′
)
∂
w
a
⁢
1
(
13
)
w
a
⁢
2
:=
w
a
⁢
2
-
β
a
⁢
∂
Ψ
⁡
(
e
′
)
∂
w
a
⁢
2
where := represents assignment operation, and βa=0.005 is a learning rate of the action network during the pre-training process;
repeat the execution of (10)-(13) until Ψ(e′)<10−5;
Step 3:
online optimization of the control strategy based on the action-critic framework;
during an online training process, an output of the critic network can be expressed as
V
^
(
e
k
,
η
^
(
e
k
)
)
=
w
c
⁢
2
T
⁢
ϕ
⁡
(
w
c
⁢
1
T
[
e
k
,
η
^
(
e
k
)
]
T
)
(
14
)
where wc1∈ and wc2∈ are weight vectors of the critic network;
hc=12 represents the number of hidden layer neurons in the critic network;
during the online training process, an approximation error of the critic network can be expressed as
ε
c
(
e
k
)
=
γ
⁢
V
^
(
e
k
,
Δ
⁢
u
k
)
-
(
V
^
(
e
k
-
1
,
Δ
⁢
u
k
-
1
)
-
U
⁡
(
e
k
,
Δ
⁢
u
k
)
)
(
15
)
a performance index of the critic network can be expressed as
E
c
(
e
k
)
=
1
2
⁢
ε
c
2
(
e
k
)
(
16
)
according to the principle of gradient descent, the update rule of the weight vector wc2 can be expressed as
w
c
⁢
2
:=
w
c
⁢
2
-
I
c
⁢
∂
E
c
(
e
k
)
∂
w
c
⁢
2
(
17
)
where lc=0.01 is the learning rate during the online training process of the critic network;
during the weight training process, only the outer-layer weight wc2 is updated, and the inner-layer weight wc1 remains unchanged after being randomly initialized;
repeat steps (14)-(17) until Ec(ek)<10−5;
at this point, update of the critic network at the k moment is completed;
next, based on evaluation results of the control strategy by the critic network, policy improvement is completed;
during the online training process, the output of the action network can be expressed as
η
ˆ
(
e
k
)
=
w
a
⁢
2
T
⁢
ϕ
⁡
(
w
a
⁢
1
T
⁢
e
k
)
(
18
)
during the online training process, an approximation error of the action network is defined as
ε
a
(
e
k
)
=
V
ˆ
(
e
k
,
Δ
⁢
u
k
)
-
U
d
(
19
)
where Ud is an ideal training target;
where Ud=0;
a performance index of the action network during the online training process is defined as
E
a
(
e
k
)
=
1
2
⁢
ε
a
2
(
e
k
)
(
20
)
an update rule of the weight vector wa2 can be expressed as
w
a
⁢
2
:=
w
a
⁢
2
-
I
a
⁢
∂
E
a
(
e
k
)
∂
w
a
⁢
2
(
21
)
repeat steps (18)-(21) until |∂Ea(ek)/∂wa2<10−3;
at this point, optimization of the action network at k moment is completed;
finally, use the well-trained action network to calculate the control input based on a current tracking error;
{
Δ
⁢
u
k
=
η
ˆ
(
e
k
)
u
k
=
u
k
⁢
1
+
Δ
⁢
u
k
(
22
)
the control input Uk represents an ideal oxygen transfer coefficient of the wastewater treatment system at a certain moment;
once a programmable logic controller (2.2) obtains the ideal value, it instantly calculates a difference between the ideal and actual oxygen transfer coefficients;
then, it generates an output signal based on this difference and sends the output signal to a frequency converter (2.3);
upon receiving the output signal, the frequency converter (2.3) responds promptly and precisely adjusts its output frequency to a blower (2.4);
by doing so, a motor speed of the blower is regulated, thereby changing the amount of air delivered to a tank (1.8) and enabling the actual oxygen transfer coefficient to reach the ideal value;
under the action of the ideal oxygen transfer coefficient, the DO concentration in the tank (1.8) is precisely regulated and stabilized near the set value, ensuring the best wastewater treatment effect.

799 more elements

Adaptive critic control algorithm integrating knowledge transfer for wastewater treatment process

Life of this patent

Adaptive critic control algorithm integrating knowledge transfer for wastewater treatment process

Life of this patent

Abstract

Claims1