# US 2025/0096600 A1 · Platform and method for power grid frequency regulation with participation of large-scale energy storage based on maddpg

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2025/0096600 A1
- Document ID: US20250096600A1
- Status: Published application
- Kind: A1
- Application number: 18/406,185
- Filed: 2024-01-07
- Published: 2025-03-20
- Earliest claimed: 2023-09-14
- Assignees: Wuhan University
- Applicants: Wuhan University
- Inventors: Jun YANG; Yuxin WEN; Peixiao FAN
- Web page: https://patentgrep.com/patent/US20250096600A1

## CPC

- H02J 13/12: Monitoring network conditions, e.g. electrical magnitudes or operational status
- H02J 3/32: using batteries or super capacitors with converting means
- H02J 2103/30: Simulating, planning, modelling, reliability check or computer assisted design [CAD] of electric power networks
- H02J 2103/35: Grid-level management of power transmission or distribution systems, e.g. load flow analysis or active network management
- H02J: ELECTRIC POWER NETWORKS; CIRCUIT ARRANGEMENTS OR SYSTEMS FOR SUPPLYING OR DISTRIBUTING ELECTRIC POWER; SYSTEMS FOR STORING ELECTRIC ENERGY

## Abstract

The present disclosure relates to the field of frequency control technologies of power grid systems and in particular to a platform and method for power grid frequency regulation with participation of large-scale energy storage based on MADDPG. Firstly, based on a scene of power grid frequency regulation with participation of large-scale energy storage, a control platform architecture for power grid frequency regulation with participation of large-scale energy storage based on MADDPG is designed. Then, based on the multiple agents deep deterministic policy gradient algorithm, under the drive of environmental interaction data, training and learning are performed to obtain optimal control multiple agents for power grid frequency regulation with participation of energy storage. Finally, by using the optimal control multiple agents, control on the output power of the thermal power unit and the charge and discharge of the large-scale energy storage is performed for participation of power grid frequency regulation.

## Claims

### Claim 1

A control platform for power grid frequency regulation with participation of large-scale energy storage based on MADDPG (Multiple Agents Deep Deterministic Policy Gradient), comprising: a three-layer architecture of a state space layer, an interaction transfer layer and a control command layer;
the state space layer comprises a power grid state monitoring apparatus, a thermal power unit and a large-scale energy storage apparatus, to achieve a real-time data monitoring and collection for a frequency deviation state of a power grid, a frequency regulation capacity of the thermal power unit, and a capacity of the energy storage apparatus;
the interaction transfer layer comprises processing data of the state space layer and transmitting observation information data of an environment for the control command layer; based on control instructions of optimal control multiple agents in the control command layer, performing a power output adjustment to the thermal power unit in the state space layer and performing a charge and discharge control on the large-scale energy storage apparatus for participation in a power grid frequency regulation;
the control command layer comprises executing, by the optimal control multiple agents, corresponding action instructions based on power grid environment information obtained from the interaction transfer layer and its own optimal control policy.

### Claim 2

A control method for power grid frequency regulation with participation of large-scale energy storage based on MADDPG (Multiple Agents Deep Deterministic Policy Gradient), comprising:
based on a MADDPG algorithm, performing a training and learning under a drive of environment interaction data to obtain optimal control multiple agents for participation of an energy storage in a power grid frequency regulation;
by using the optimal control multiple agents, performing a control on an output power of an thermal power unit and on a charge and discharge of a large-scale energy storage for participation in a power grid frequency regulation.

### Claim 3 (depends on 2)

The control method of claim 2, wherein, based on the MADDPG algorithm, performing the training and learning under the drive of the environment interaction data to obtain the optimal control multiple agents for participation of the energy storage in the power grid frequency regulation comprises following steps:
step 1) based on a scene of a primary power grid frequency regulation with participation of the large-scale energy storage, it is described as a Markov game process, comprising following components:
(1-1) a number N of multiple agents: in a frequency regulation scene, determining a corresponding number of agents based on a number of control apparatuses;
(1-2) a state space S: state information to be obtained by the multiple agents in a control process;
(1-3) an action space A: executing corresponding control actions by the multiple agents based on current state information and its own policy in the control process;
(1-4) a reward function R: after sending control action instructions, the multiple agents obtain rewards fed back from an environment to determine a superiority of a current policy;
(1-5) a discount factor γ: representing an influence degree of a future reward on a current;
step 2) based on the scene of the power grid frequency regulation with participation of the energy storage, the state space, the action space and the reward function are determined; each agent obtains from a power grid the state information, comprising a load fluctuation and a frequency deviation of the power grid, upper and lower limit constraints of a frequency regulation capacity of the thermal power unit, and upper and lower limit constraints of charge and discharge of each energy storage apparatus, wherein the state information is integrated into the state space with expression in formula (1):
S
=
[
Δ
⁢
f
⁡
(
t
)
,
P
g
+
(
t
)
,
P
g
-
(
t
)
,
P
bes
,
1
+
(
t
)
,
P
bes
,
1
-
(
t
)
,
…
,
P
bes
,
k
+
(
t
)
,
P
bes
,
k
-
(
t
)
]
formula
⁢
(
1
)
wherein Δf(t) is the frequency deviation of the power grid at a moment t, Pg+(t) and Pg−(t) are upper and lower capacity limits for a participation of the thermal power unit in the frequency regulation at the moment t, and Pbes,k+(t) and Pbes,k−(t) are upper and lower charge and discharge limits of the energy storage apparatus k at the moment t;
setting of the number of the multiple agents: one agent is set for frequency regulation apparatuses respectively, and (1+k) agents corresponding to one thermal power unit and k energy storage apparatuses are set; corresponding to different agents, the state space is selected in formula (1) based on own circumstances, the agents of the thermal power unit obtain the frequency deviation and the state information of the upper and lower limits of the thermal power unit; and the agents of the energy storage apparatus obtain the frequency deviation and the state information of the upper and lower charge and discharge limits of the energy storage apparatus;
setting of the action space: the action space of an entire power grid frequency regulation comprises a set of control actions of different agents for the thermal power unit and the energy storage apparatus, with its expression shown in below formula (2):
A
=
[
A
g
,
A
1
,
…
,
A
k
]
formula
⁢
(
2
)
wherein
,
{
A
g
=
Δ
⁢
P
g
(
t
)
A
k
=
P
bes
,
k
(
t
)
formula
⁢
(
3
)
wherein ΔPg(t) represents a power output adjustment amount of the thermal power unit at the moment t, and ΔPbes,k(t) represents a charge and discharge power of the energy storage apparatus k at the moment t;
setting of the reward function: an expression of the reward function is below formula (4):
r
g
=
{
0
❘
"\[LeftBracketingBar]"
Δ
⁢
f
❘
"\[RightBracketingBar]"
<
0.03
-
μ
1
⁢
❘
"\[LeftBracketingBar]"
Δ
⁢
f
❘
"\[RightBracketingBar]"
0.03
≤
❘
"\[LeftBracketingBar]"
Δ
⁢
f
❘
"\[RightBracketingBar]"
<
0.08
-
μ
2
⁢
❘
"\[LeftBracketingBar]"
Δ
⁢
f
❘
"\[RightBracketingBar]"
0.08
≤
❘
"\[LeftBracketingBar]"
Δ
⁢
f
❘
"\[RightBracketingBar]"
<
0.12
-
μ
3
⁢
❘
"\[LeftBracketingBar]"
Δ
⁢
f
❘
"\[RightBracketingBar]"
0.12
≤
❘
"\[LeftBracketingBar]"
Δ
⁢
f
❘
"\[RightBracketingBar]"
<
0.16
-
μ
4
⁢
❘
"\[LeftBracketingBar]"
Δ
⁢
f
❘
"\[RightBracketingBar]"
0.16
≤
❘
"\[LeftBracketingBar]"
Δ
⁢
f
❘
"\[RightBracketingBar]"
<
0.2
-
μ
5
⁢
❘
"\[LeftBracketingBar]"
Δ
⁢
f
❘
"\[RightBracketingBar]"
0.2
≤
❘
"\[LeftBracketingBar]"
Δ
⁢
f
❘
"\[RightBracketingBar]"
formula
⁢
(
4
)
wherein rg represents the reward function of the agent of the thermal power unit, r μ1, μ2, μ3, μ4 and μ5 represent weight values corresponding to the reward functions of control intervals of the frequency deviation; when the frequency deviation is in an adjustment dead zone [0, 0.03] Hz, the frequency deviation is far less than a minimum error requirement for a normal operation and a reward value of the agent is set to 0; when the frequency deviation is in different set intervals, an agent controller obtains a corresponding negative reward;
r
bes
=
{
0
❘
"\[LeftBracketingBar]"
Δ
⁢
f
❘
"\[RightBracketingBar]"
<
0.04
-
δ
1
⁢
❘
"\[LeftBracketingBar]"
Δ
⁢
f
❘
"\[RightBracketingBar]"
0.04
≤
❘
"\[LeftBracketingBar]"
Δ
⁢
f
❘
"\[RightBracketingBar]"
<
0.08
-
δ
2
⁢
❘
"\[LeftBracketingBar]"
Δ
⁢
f
❘
"\[RightBracketingBar]"
0.08
≤
❘
"\[LeftBracketingBar]"
Δ
⁢
f
❘
"\[RightBracketingBar]"
<
0.12
-
δ
3
⁢
❘
"\[LeftBracketingBar]"
Δ
⁢
f
❘
"\[RightBracketingBar]"
0.12
≤
❘
"\[LeftBracketingBar]"
Δ
⁢
f
❘
"\[RightBracketingBar]"
<
0.16
-
δ
4
⁢
❘
"\[LeftBracketingBar]"
Δ
⁢
f
❘
"\[RightBracketingBar]"
0.16
≤
❘
"\[LeftBracketingBar]"
Δ
⁢
f
❘
"\[RightBracketingBar]"
<
0.2
-
δ
5
⁢
❘
"\[LeftBracketingBar]"
Δ
⁢
f
❘
"\[RightBracketingBar]"
0.2
≤
❘
"\[LeftBracketingBar]"
Δ
⁢
f
❘
"\[RightBracketingBar]"
formula
⁢
(
5
)
wherein rbes represents the reward function of the agent of the energy storage apparatus; and δ1, δ2, δ3, δ4 and δ5 represent the weight values corresponding to the reward functions of the control intervals of the frequency deviation;
step 3) the multiple agents deep deterministic policy gradient algorithm has following flow:
(3-1) setting a total number N of cyclic trainings and initializing an experience replay pool D and a multiple agents policy network π(St|θ);
(3-2) setting a round number M of trainings;
(3-3) initializing a current environment state S0;
(3-4) for each agent, based on environment state information of a current moment, generating random actions superimposed with a Gaussian distribution noise to perform an interactive exploration with a power grid environment, with the expression in below formula (6):
A
=
π
⁡
(
S
t
|
θ
)
+
λ
formula
⁢
(
6
)
wherein θ represents a policy network parameter of multiple agents, and λ represents the Gaussian distribution noise;
(3-5) based on action instructions executed by the multiple agents, returning, by the power grid environment, the corresponding reward function Rt and transferring to a state St+1 of a next moment;
(3-6) storing the information of (St, At, Rt, St+1) to the experience replay pool;
(3-7) when information stored in the experience replay pool reaches a specified amount Dx, performing an update of the multiple agents based on randomly-selected batch data, calculating a corresponding loss function J and using a MSE (Mean Square Error) method to perform a gradient update of a policy network, with its expression in below formula (7):
J
=
1
N
⁢
∑
j
(
y
j
-
Q
(
s
j
,
π
⁡
(
s
j
+
1
⁢
❘
"\[LeftBracketingBar]"
θ
)
❘
"\[RightBracketingBar]"
⁢
θ
critic
)
)
2
formula
⁢
(
7
)
wherein
,
y
j
=
r
j
+
1
+
γ
⁢
Q
(
s
j
+
1
,
π
⁡
(
s
j
+
1
⁢
❘
"\[LeftBracketingBar]"
θ
-
)
❘
"\[RightBracketingBar]"
⁢
θ
critic
-
)
formula
⁢
(
8
)
wherein Q represents a state action value function, θ and θ− respectively represent parameters of the policy network and its target network, θcritic and θcritic− respectively represent parameters of an evaluation network and its target network;
(3-8) after training rounds are completed, updating a network parameter with the expression in below formula (9):
{
θ
-
=
τ
⁢
θ
+
(
1
-
τ
)
⁢
θ
-
θ
critic
-
=
τ
⁢
θ
critic
+
(
1
-
τ
)
⁢
θ
critic
-
formula
⁢
(
9
)
wherein τ is an update parameter;
(3-9) after cyclic trainings are completed, obtaining an optimal control multiple agents model;
4. by performing a large number of interactions with an environment, the multiple agents continuously optimizes and updates deep neural network parameters under a guidance of a loss function optimization algorithm, and after the trainings are completed, obtains an optimal control multiple agents policy and takes the policy as a control command layer.

### Claim 4 (depends on 2)

The control method of claim 2, wherein by using the optimal control multiple agents, performing the control on the output power of the thermal power unit and on the charge and discharge of the large-scale energy storage for participation in the power grid frequency regulation comprises following smart control process:
an upper control command layer collects power grid environment information from a state space layer by an interaction transfer layer, wherein the power grid environment information comprises a frequency deviation, upper and lower limit constraints of the thermal power unit, and upper and lower limit constraints of charge and discharge of an energy storage apparatus; based on an optimal control multiple agents model, an optimal control policy is used and a corresponding action instruction is given, and an interaction transfer layer, based on the upper-layer action instruction, performs a power output control on the thermal power unit and a charge and discharge control on an energy storage apparatus in the lower state space layer, so as to achieve optimal control on a primary power grid frequency regulation with participation of the energy storage.

### Claim 5 (depends on 2)

An electronic device, comprising a memory, a processor and a computer program stored in the memory and run on the processor, wherein the processor executes the program to perform the control method for power grid frequency regulation with participation of large-scale energy storage based on MADDPG according to claim 2.

### Claim 6 (depends on 2)

A non-transient computer readable storage medium, storing a computer program, wherein the computer program is executed by a processor to perform the control method for power grid frequency regulation with participation of large-scale energy storage based on MADDPG according to claim 2.

### Claim 7 (depends on 2)

A computer program product, comprising a computer program, wherein the computer program is executed by a processor to perform the control method for power grid frequency regulation with participation of large-scale energy storage based on MADDPG according to claim 2.

### Claim 8 (depends on 3)

An electronic device, comprising a memory, a processor and a computer program stored in the memory and run on the processor, wherein the processor executes the program to perform the control method for power grid frequency regulation with participation of large-scale energy storage based on MADDPG according to claim 3.

### Claim 9 (depends on 4)

An electronic device, comprising a memory, a processor and a computer program stored in the memory and run on the processor, wherein, the processor executes the program to perform the control method for power grid frequency regulation with participation of large-scale energy storage based on MADDPG according to claim 4.

### Claim 10 (depends on 3)

A non-transient computer readable storage medium, storing a computer program, wherein the computer program is executed by a processor to perform the control method for power grid frequency regulation with participation of large-scale energy storage based on MADDPG according to claim 3.

### Claim 11 (depends on 4)

A non-transient computer readable storage medium, storing a computer program, wherein the computer program is executed by a processor to perform the control method for power grid frequency regulation with participation of large-scale energy storage based on MADDPG according to claim 4.

### Claim 12 (depends on 3)

A computer program product, comprising a computer program, wherein the computer program is executed by a processor to perform the control method for power grid frequency regulation with participation of large-scale energy storage based on MADDPG according to claim 3.

### Claim 13 (depends on 4)

A computer program product, comprising a computer program, wherein the computer program is executed by a processor to perform the control method for power grid frequency regulation with participation of large-scale energy storage based on MADDPG according to claim 4.
