# US 11,442,420 B2 · Power grid reactive voltage control method based on two-stage deep reinforcement learning

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,442,420 B2
- Document ID: US11442420B2
- Status: Granted
- Kind: B2
- Application number: 17/026,364
- Filed: 2020-09-21
- Published: 2021-11-18
- Granted: 2022-09-13
- Earliest claimed: 2020-09-21
- Assignees: Tsinghua University
- Applicants: Tsinghua University
- Inventors: Wenchuan Wu; Haotian Liu; Hongbin Sun; Bin Wang; Qinglai Guo; Tian Xia
- Web page: https://patentgrep.com/patent/US11442420B2

## CPC

- G05B 19/042: using digital processors (G05B19/05 takes precedence)
- H02J 3/16: by adjustment of reactive power
- G06N 3/006: based on simulated virtual individual or collective life forms, e.g. social simulations or particle swarm optimisation [PSO]
- G06N 7/01: Probabilistic graphical models, e.g. probabilistic networks
- G06N 20/00: Machine learning
- G05B 2219/2639: Energy management, use maximum of cheap power, keep peak load low
- H02J 3/1864: using reactive elements connected in series with semiconductor switches, e.g. static VAR compensators [SVC], thyristor-controlled reactors [TCR] or thyristor-switched capacitors [TSC]
- H02J 3/381: {Dispersed generators}
- H02J 2103/30: Simulating, planning, modelling, reliability check or computer assisted design [CAD] of electric power networks
- H02J 2103/35: Grid-level management of power transmission or distribution systems, e.g. load flow analysis or active network management
- Y02B 10/10: Photovoltaic [PV]
- Y02E 40/70: Smart grids as climate change mitigation technology in the energy generation sector
- Y02E 60/00: Enabling technologies; Technologies with a potential or indirect contribution to GHG emissions mitigation
- Y04S 10/50: Systems or methods supporting the power network operation or management, involving a certain degree of interaction with the load-side end user applications
- Y04S 40/20: Information technology specific aspects, e.g. CAD, simulation, modelling, system security
- G05B: CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS (systems for controlling or regulating non-electric variables G05D; systems for regulating electric or magnetic variables G05F; control devices or systems insofar as characterised by mechanical features only G05G)
- H02J: ELECTRIC POWER NETWORKS; CIRCUIT ARRANGEMENTS OR SYSTEMS FOR SUPPLYING OR DISTRIBUTING ELECTRIC POWER; SYSTEMS FOR STORING ELECTRIC ENERGY
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- Y02B: CLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO BUILDINGS, e.g. HOUSING, HOUSE APPLIANCES OR RELATED END-USER APPLICATIONS
- Y02E: REDUCTION OF GREENHOUSE GAS [GHG] EMISSIONS, RELATED TO ENERGY GENERATION, TRANSMISSION OR DISTRIBUTION
- Y04S: SYSTEMS INTEGRATING TECHNOLOGIES RELATED TO POWER NETWORK OPERATION, COMMUNICATION OR INFORMATION TECHNOLOGIES FOR IMPROVING THE ELECTRICAL POWER GENERATION, TRANSMISSION, DISTRIBUTION, MANAGEMENT OR USAGE, i.e. SMART GRIDS

## Abstract

A power grid reactive voltage control method and control system based on two-stage deep reinforcement learning, comprising steps of: building interactive training environment based on Markov decision process, according to a regional power grid simulation model and a reactive voltage optimization model; training a reactive voltage control model offline by using a SAC algorithm, in the interactive training environment based on Markov decision process; deploying the reactive voltage control model to a regional power grid online system; and acquiring operating state information of the regional power grid, updating the reactive voltage control model, and generating an optimal reactive voltage control policy. As compared with the existing power grid optimizing method based on reinforcement learning, the online control training according to the present disclosure has costs and safety hazards greatly reduced, and is more suitable for deployment in an actual power system.

## Claims

### Claim 1

A power grid reactive voltage control method based on two-stage deep reinforcement learning, comprising the steps of:
building, by a regional power grid control center server, interactive training environment based on Markov decision process, according to a regional power grid simulation model and a reactive voltage optimization model;
training, by the regional power grid control center server, a reactive voltage control model offline by using SAC algorithm, in the interactive training environment based on Markov decision process;
deploying, by the regional power grid control center server, the reactive voltage control model to a regional power grid controller of a regional power grid online system; and
acquiring, by the regional power grid controller, operating state information of the regional power grid, updating the reactive voltage control model, and generating an optimal reactive voltage control policy.

### Claim 2 (depends on 1)

The power grid reactive voltage control method based on two-stage deep reinforcement learning according to claim 1, further comprising the steps of:
sending, by the regional power grid controller, the optimal reactive voltage control policy to respective controllable devices, and re-acquiring, by the regional power grid controller, operating state information of the regional power grid,
wherein, the respective controllable devices include distributed generation devices and static var compensators provided on the respective nodes of the regional power grid.

### Claim 3 (depends on 2)

The power grid reactive voltage control method based on two-stage deep reinforcement learning according to claim 2, wherein, sending the optimal reactive voltage control policy to the respective controllable devices, and re-acquiring operating state information of the regional power grid, includes:
issuing the optimal reactive voltage control policy to respective corresponding devices through a power grid remote control system;
re-acquiring regional power grid operating state information s′t, calculating a feedback variable value rt, and updating an experience library as
D←D∪{(st,at,rt,s′t)};
wherein, the experience library D contains the state variable st of the regional power grid at time t, an optimal control policy at, the feedback variable rt, and the state variable s′t of the regional power grid at time t+1;
repeating the acquiring operating state information of the regional power grid, updating the reactive voltage control model, and generating an optimal reactive voltage control policy.

### Claim 4 (depends on 1)

The power grid reactive voltage control method based on two-stage deep reinforcement learning according to claim 1, further comprising constructing the regional power grid simulation model by the regional power grid control center server,
wherein, constructing the regional power grid simulation model includes:
determining an undirected graph model Π(N,E) of the regional power grid, according to relative positions between n+1 nodes in the regional power grid, where, N=0, . . . , n, which is a set of the nodes, and E=(i,j)∈N×N, which is a set of the branches;
constructing a power flow calculation model of the regional power grid:
Pij=GijVi2−GijViVj cos θij−BijViVj sin θij,∀ij∈E
Qij=−BijVi2+BijViVj cos θij−GijViVj sin θij,∀ij∈E
θij=θi−θj,∀ij∈E,
wherein, Vi,θi are a voltage amplitude and a phase angle of node i; Gij,Bij are conductance and susceptance of branch ij; Pij,Qij are active power and reactive power of branch ij; and θij is a phase angle difference of branch ij;
constructing a node power model of the regional power grid:
P
j
=
G
sh
,
i
⁢
V
i
2
+
∑
j
∈
K
⁡
(
i
)
⁢
P
i
⁢
j
=
{
-
P
Dj
,
j
∈
N
∖
N
I
⁢
B
P
Gj
-
P
Dj
,
j
∈
N
∖
N
I
⁢
B
⁢
⁢
Q
j
=
-
B
sh
,
i
⁢
V
i
2
+
∑
j
∈
K
⁡
(
i
)
⁢
Q
i
⁢
j
=
{
-
Q
Dj
,
j
∈
N
∖
{
N
I
⁢
B
⋃
N
C
⁢
D
}
Q
G
⁢
j
-
Q
D
⁢
j
,
j
∈
N
I
⁢
B
Q
C
⁢
j
-
Q
Dj
,
j
∈
N
C
⁢
D
,
wherein, Pj,Qj are active power injection and reactive power injection of node j; Gsh,i,Bsh,i are respectively ground conductance and susceptance of node i; PDj,QDj are active power load and reactive power load of node j; QGj is DG reactive power output of node j; QCj is static var compensator reactive power output of node j; NIB is a set of nodes coupled to DG, and NCD is a set of nodes coupled to static var compensators.

### Claim 5 (depends on 1)

The power grid reactive voltage control method based on two-stage deep reinforcement learning according to claim 1, further comprising constructing the reactive voltage optimization model by the regional power grid control center server,
wherein, the reactive voltage optimization model includes:
min
⁢
∑
j
∈
N
⁢
P
j
s
.
t
.
⁢
V
i
_
≤
V
i
≤
V
i
_
,
∀
i
∈
N

Q
G
⁢
i

≤
S
G
⁢
i
2
-
P
G
⁢
i
_
2
,
∀
i
∈
N
Q
C
⁢
i
_
≤
Q
Ci
≤
Q
Ci
_
,
∀
i
∈
N
,
wherein, Vi,Vi are a lower limit and an upper limit of a voltage of node i; QCi,QCi are a lower limit and an upper limit of SVC reactive power output of node i; and SGi,PGi are DG installed capacity and an active power output upper limit of node i.

### Claim 6 (depends on 1)

The power grid reactive voltage control method based on two-stage deep reinforcement learning according to claim 1, wherein, building interactive training environment based on Markov decision process, according to a regional power grid simulation model and a reactive voltage optimization model, includes:
acquiring the operating state information of the regional power grid according to measured signals of the regional power grid, and constructing a Markov decision process state variable
s=(P,Q,V,t),
wherein, P, Q are node active power and reactive power injection vectors; V is a node voltage vector; and t is a time variable during training;
constructing a feedback variable, according to the reactive voltage optimization model
rt=−Σi∈NPi(t)−CVΣi∈N[ReLU2(Vi(t)−V)+ReLU2(V−Vi(t))],
wherein, CV is a voltage suppression coefficient; and ReLU is a non-linear function, ReLU(x)=max(0,x);
determining an action variable, according to reactive power of controllable flexible resources
a=(QG,QC),
wherein, QG,QC are respectively reactive power output vectors of respective distributed generation devices and static var compensators.

### Claim 7 (depends on 1)

The power grid reactive voltage control method based on two-stage deep reinforcement learning according to claim 1, wherein, training a reactive voltage control model offline by using a SAC algorithm, includes:
constructing a reinforcement learning target function
J=Σt=0∞γt(rt+αH(π(·|st))),
wherein, γ is a reduction coefficient; α is a maximum entropy multiplier; H is an entropy function; and π(·|st) is a policy function;
converting form of the policy function, by using reparameterization trick,
ãθ(s,ξ)=tan h(μθ(s)+σθ(s)⊙ξ),ξ˜N(0,I),
wherein, θ is a policy network parameter; μθ and σθ are a mean value and a variance function corresponding thereto; and N(0,I) is a standard Gaussian distribution function;
defining and training a value function network model Qπ(s,a);
training a policy network model
max
θ
⁢
E
s
∼
𝒟
ξ
∼
N
⁡
[
Q
⁡
(
s
,
a
~
θ
⁢
⁢
(
s
,
ξ
)
)
-
αlog
⁢
π
θ
⁢
(
a
~
θ
⁡
(
s
,
ξ
)
|
s
)
]
.

### Claim 8 (depends on 7)

The power grid reactive voltage control method based on two-stage deep reinforcement learning according to claim 7, wherein, defining and training a value function network Qπ(s,a) includes steps of:
obtaining a recursive form of Qπ(s,a) through a Bellman equation
Q
π
⁡
(
s
,
a
)
=
E
s
′
,
a
′
⁡
[
R
⁡
(
s
,
a
,
s
′
)
+
γ
⁡
(
Q
π
⁡
(
s
′
,
a
′
)
+
α
⁢
⁢
H
⁡
(
π
⁡
(
·
|
s
′
)
)
)
]
wherein, s is a state variable at time t; s′ is a state variable at time t+1; a is an action variable at time t; and a′ is an action variable at time t+1;
calculating an estimated value of the value function network Qπ(s,a)
y=r+γ(Qπ(s′,ã′)−α log π(ã′|s′)),ã′˜π(·|s′);
training the value function network, according to the estimated value y of the value function network Qπ(s,a)
min(Qπ(s′,a′)−y)2,ã′˜π(·|s′).

### Claim 9 (depends on 1)

The power grid reactive voltage control method based on two-stage deep reinforcement learning according to claim 1, comprising, deploying the reactive voltage control model to the regional power grid controller of the regional power grid online system by the regional power grid control center server, initializing, by the regional power grid controller the time variable t, and initializing the experience library D.

### Claim 10 (depends on 9)

The power grid reactive voltage control method based on two-stage deep reinforcement learning according to claim 9, wherein, acquiring operating state information of the regional power grid, updating the reactive voltage control model, and generating an optimal reactive voltage control policy, includes steps of:
acquiring a measured signal of the regional power grid at time t, and forming a corresponding state variable st=(P,Q,V,t), wherein, P, Q are an active power injection vector and a reactive power injection vector of nodes, V is a node voltage vector, and t is a time variable during training;
extracting a set of experiences from the experience library D, DB∈D, with a quantity of B;
updating the reactive voltage control model on DB, by using the value function network and the policy network trained;
generating an optimal policy at time t, by using the updated reactive voltage control model
at=tan h(μθ(st)+σθ(st)⊙ξ)=(QG,QC), wherein, θ is a policy network parameter; μ0 and σθ are a mean value and a variance function corresponding thereto, ξ is a random variable, QG,QC are respectively reactive power output vectors of the respective distributed generation devices and static var compensators.

### Claim 11

A power grid reactive voltage control system based on two-stage deep reinforcement learning, comprising:
a training environment building module, configured to build interactive training environment based on Markov decision process, according to a regional power grid simulation model and a reactive voltage optimization model;
a training module, configured to train a reactive voltage control model offline by using a SAC algorithm;
a transferring module, configured to deploy the reactive voltage control model to a regional power grid online system; and
a policy generating module, configured to acquire operating state information of the regional power grid, update the reactive voltage control model, and generate an optimal reactive voltage control policy.

### Claim 12 (depends on 11)

The power grid reactive voltage control system based on two-stage deep reinforcement learning according to claim 11, further comprising:
a continuous online learning module, configured to send the optimal reactive voltage control policy to respective controllable devices, and re-acquire operating state information of the regional power grid.

### Claim 13 (depends on 11)

The power grid reactive voltage control system based on two-stage deep reinforcement learning according to claim 11, further comprising:
a simulation model constructing module, configured to construct the regional power grid simulation model; and
a reactive voltage optimization model constructing module, configured to construct the reactive voltage optimization model of the regional power grid, according to a reactive voltage control target of the regional power grid.