US 12,755,217 B2Granted

System and method for a hierarchical multi-agent framework for transactive microgrids

AssigneeMohamed bin Zayed University of Artificial Intelligence

InventorsMartin Takáč, Nicolás Mauricio Cuadrado Ávila, Roberto Alejandro Gutiérrez Guillén, Samuel Horvath

Life of this patent

Estimating the term…

Loading patent…

Abstract

A multi-agent reinforcement learning framework for managing energy transactions in microgrids that includes three layers of agents, each pursuing different objectives. The first layer, including prosumers and consumers, minimizes the total energy cost. The other two layers control the energy price to decrease the carbon emission impact while balancing the consumption and production of both renewable and conventional energy. The framework takes into account fluctuations in energy demand and supply due to household supplied energy from renewable energy sources and energy storage levels in household energy storage devices.

Claims17

Dependent claims

A hierarchical transactive energy control system for controlling a plurality of microgrids, comprising:

a distributor reinforcement learning agent distributing electric power among the plurality of microgrids by facilitating energy trading among the microgrids in a manner that takes into account carbon emission by setting buy and sell prices among the plurality of microgrids;
a plurality of microgrid reinforcement learning agents for each of the plurality of microgrids, wherein each microgrid agent controls a plurality of household electric loads and is configured to access energy available at other microgrids or provide energy to said other microgrids, wherein the plurality of microgrid reinforcement learning agents set a sell price and a buy price for the plurality of household electric loads in respective microgrids which are controlled by a respective said microgrid reinforcement learning agents;
and
a plurality of household reinforcement learning agents for controlling each of the plurality of household electric loads, wherein at least one active consumer household includes an energy storage device, wherein the household reinforcement learning agent for the at least one consumer household determines how to optimize energy usage from the energy storage device;
wherein each of said household reinforcement learning agents is a reinforcement machine learning model having an actor and a critic with an advantage function, in which a value V is subtracted from an expected reward, for doing an action in a given state, in order to re-calibrate the expected reward towards an average action,
wherein the reward takes into account the carbon emission,
wherein the action is to charge or discharge the battery,
wherein the state is the state of charge of the battery.

5 more elements

A method for transactive energy control in a hierarchical multi-agent control system, the hierarchical multi-agent control system comprising a household layer, a microgrid layer, and a distributor layer, wherein the household layer includes a plurality of household reinforcement learning agents, the microgrid layer includes a plurality of microgrid reinforcement learning agents, and the distributor layer includes a distributor reinforcement learning agent, the method comprising:

controlling, by each household reinforcement learning agent, a household electric load and charging and discharging of respective household batteries,
training each household reinforcement learning agent as an actor-critic reinforcement learning model, in which a critic neural network evaluates actions of an actor neural network and approximates states of the actor neural network,
wherein the actions include the charging and discharging the respective household batteries,
wherein the states include state of charge of the household batteries;
when energy in the household layer is in a shortage state, importing energy from an external power grid, and when energy in the household layer is in a surplus state, exporting energy to the external power grid, in a manner that energy imported and energy exported is minimized;
maximizing, by a microgrid reinforcement learning agent, use of local energy in the microgrid based on a pricing policy for local transactions,
when a microgrid is in an energy shortage state such that its local energy is insufficient to cover internal demand, accessing energy in other microgrids,
when a microgrid is in an energy surplus state such that distributed generation surpasses the internal demand, selling energy to other microgrids experiencing a shortage,
when energy is unavailable at the microgrid layer, importing energy from the distributor layer,
when the energy is over-produced at the microgrid layer, exporting energy to the distributor layer;
setting, by the distributor reinforcement learning agent, buy and sell prices among the microgrids in a manner that simultaneously facilitates energy trading among the microgrids and minimizes carbon emissions.

8 more elements

controlling, by each household reinforcement learning agent, a household electric load and charging and discharging of respective household batteries,
when energy in the household layer is in a shortage state, importing energy from an external power grid, and when energy in the household layer is in a surplus state, exporting energy to the external power grid, in a manner that energy imported and energy exported is minimized:
maximizing, by a microgrid reinforcement learning agent, use of local energy in the microgrid based on a pricing policy for local transactions,
when a microgrid is in an energy shortage state such that its local energy is insufficient to cover internal demand, accessing energy in other microgrids,
when a microgrid is in an energy surplus state such that distributed generation surpasses the internal demand, selling energy to other microgrids experiencing a shortage,
when energy is unavailable at the microgrid layer, importing energy from the distributor layer,
when the energy is over-produced at the microgrid layer, exporting energy to the distributor layer;
setting, by the distributor reinforcement learning agent, buy and sell prices among the microgrids in a manner that simultaneously facilitates energy trading among the microgrids and minimizes carbon emissions:
wherein each said household reinforcement learning agent is a reinforcement machine learning model having an actor and a critic with an advantage function,
subtracting a value V from an expected reward, for doing an action in a given state, in order to re-calibrate the expected reward towards an average action,
wherein the reward takes into account the carbon emission,
wherein the action is to charge or discharge a household battery,
wherein the state is the state of charge of the household battery.

10 more elements

Dependencies are inferred from the claim text.

Facts

Application: 18/587,182Filed 26 Feb 2024
Published: US 2025/0272621 A128 Aug 2025
Granted: 6 Oct 2026
Term ends: Not estimatedThe source records don’t support an estimate.

Classification3 codes

G06Q 10/04
Administration; Management: forecasting or optimisation specially adapted for administrative or management purposes, e.g. linear programming or…
G06Q 50/06
Information and communication technology [ICT] specially adapted for implementation of business processes of specific business sectors, e.g.…:…
G06Q
Information and communication technology [ict] specially adapted for administrative, commercial, financial, managerial