# US 12,111,620 B2 · Multi-agent deep reinforcement learning for dynamically controlling electrical equipment in buildings

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,111,620 B2
- Document ID: US12111620B2
- Status: Granted
- Kind: B2
- Application number: 17/029,788
- Filed: 2020-09-23
- Published: 2021-07-01
- Granted: 2024-10-08
- Earliest claimed: 2019-12-13
- Assignees: Tata Consultancy Services Limited
- Applicants: Tata Consultancy Services Limited
- Inventors: Srinarayana Nagarathinam; Avinash Achar; Arunchandar Vasan
- Web page: https://patentgrep.com/patent/US12111620B2

## CPC

- G05B 13/027: {using neural networks only}
- F24F 11/62: characterised by the type of control or by internal processing, e.g. using fuzzy logic, adaptive control or estimation of values
- G05B 15/02: electric
- G06N 3/088: Non-supervised learning, e.g. competitive learning
- G05B 2219/2614: HVAC, heating, ventillation, climate control
- G05B: CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS (systems for controlling or regulating non-electric variables G05D; systems for regulating electric or magnetic variables G05F; control devices or systems insofar as characterised by mechanical features only G05G)
- F24F: AIR-CONDITIONING; AIR-HUMIDIFICATION; VENTILATION; USE OF AIR CURRENTS FOR SCREENING (removing dirt or fumes from areas where they are produced B08B15/00; vertical ducts for carrying away waste gases from buildings E04F17/02; tops for chimneys or ventilating shafts, terminals for flues F23L17/02)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

Reinforcement Learning agent interacting with a real-world building to determine optimal policy may not be viable due to comfort constraints. Embodiments of the present disclosure provide multi-deep agent RL for dynamically controlling electrical equipment in buildings, wherein a simulation model is generated using design specification of (i) controllable electrical equipment (or subsystem) and (ii) building. Each RL agent is trained using simulation model and deployed in the subsystem. Reward function for each subsystem includes some portion of reward from other subsystem(s). Based on reward function of each RL agent, each RL agent learns an optimal control parameter during execution of RL agent in subsystem. Further, a global optimal control parameter list is generated using the optimal control parameter. The control parameters in the global optimal control parameters list are fine-tuned to improve subsystem's performance. Information on fine-tuning parameters of the subsystem and reward function are used for training RL agents.

## Claims

### Claim 1

A processor implemented method, comprising:
obtaining, via one or more hardware processors, input data comprising (i) a design specification of one or more controllable electrical equipment installed and operating in a building and (ii) design details of the building associated thereof;
generating, via the one or more hardware processors, a simulation model using the input data, wherein the simulation model is built in offline by abstracting complex Heating, ventilation, and air conditioning (HVAC) loops of the building into three loops wherein the three loops are a primary chilled water loop, a secondary chilled water loop, and an air loop, and an energy consumption of the primary loop is Ech, an energy consumption of the secondary loop is Epump, an energy consumption of the air loop is Efan;
training, via the one or more hardware processors, a plurality of deep Reinforcement learning (RL) agents using the generated simulation model, wherein the plurality of deep RL agents interacts with an environment of the building and learns to act using a Markov Decision Process (MDP) method;
deploying, via the one or more hardware processors, each of the plurality of trained deep RL agents in the one or more controllable electric equipment of the building, wherein
the primary chilled water loop is assigned with a first deep trained RL agent of the plurality of trained deep RL agents,
the secondary chilled water loop is assigned with a second deep trained RL agent of the plurality of trained deep RL agents, and
the air loop is assigned with a third deep trained RL agent of the plurality of trained deep RL agents,
each of the plurality of trained deep RL agents monitors one or more states affecting performance of the one or more controllable electric equipment in the building,
during an execution of each of the plurality of trained RL agents, a reward function is assigned to each of the plurality of trained RL agents,
the reward function comprises an energy component of the one or more controllable electrical equipment and a penalty,
the reward function comprises an energy component, and
the penalty includes a visual feedback of an associated controllable electrical equipment and degradation information of the associated controllable electrical equipment,
the reward function of the first deep trained RL agent is r1=Ech+a factor*the visual feedback,
the reward function of the second deep trained RL agent is r2=Epump+the factor*Ech, and
the reward function of the third deep trained RL agent is r3=Efan+a factor1*the visual feedback+a factor2*Ech;
triggering, via the one or more hardware processors, each of the plurality of trained deep RL agents, to obtain a portion of the reward function associated with another deep RL agent, wherein
a first trained deep RL agent of the plurality of trained deep RL agents obtains a first specific portion of the portion of the reward function from a second trained deep RL agent of the plurality of trained deep RL agents,
the first specific portion of the reward function includes the energy component only, the penalty only or combination of both the energy component and the penalty,
the first specific portion of the reward function include the energy component and the penalty,
the first trained deep RL agent is associated with a first controllable electrical equipment of the one or more controllable electrical equipment,
the second trained deep RL agent is associated with a second controllable electrical equipment of the one or more controllable electrical equipment, and
information on the portion of the reward function is utilized for training the plurality of trained deep RL agents;
estimating, via the one or more hardware processors, based on the obtained portion of the reward function, a global optimal control parameter list based on an optimal control parameter associated with each of the plurality of trained deep RL agents, wherein the optimal control parameter is learnt by each of the plurality of trained deep RL agents during an execution of the plurality of trained deep RL agents deployed in the one or more controllable electric equipment in the building, wherein the optimal control parameter comprises scheduling information of the associated controllable electrical equipment; and
fine tuning a plurality of control parameters of the global optimal control parameter list to improve a performance of each of the plurality of trained deep RL agents.

### Claim 2 (depends on 1)

The processor implemented method of claim 1, wherein the one or more controllable electrical equipment comprises one of one or more heating equipment, a ventilation system, and air conditioning (HVAC) subsystems, one or more lighting equipment, computing loads systems, or combinations thereof.

### Claim 3 (depends on 1)

The processor implemented method as claimed in claim 1, wherein the one or more states comprise at least one of a temperature, a humidity, one or more ambient parameters, a lighting intensity, and an occupant density.

### Claim 4 (depends on 1)

The processor implemented method as claimed in claim 1, wherein the penalty further comprises a thermal feedback.

### Claim 5 (depends on 1)

The processor implemented method as claimed in claim 1, wherein the optimal control parameter comprises a set point temperature, and a lighting intensity set point.

### Claim 6

A system, comprising:
a memory storing instructions;
one or more communication interfaces; and
one or more hardware processors coupled to the memory via the one or more communication interfaces, wherein the one or more hardware processors are configured by the instructions to:
obtain input data comprising (i) a design specification of one or more controllable electrical equipment installed and operating in a building and (ii) design details of the building associated thereof;
generate a simulation model using the input data, wherein the simulation model is built in offline by abstracting complex Heating, ventilation, and air conditioning (HVAC) loops of the building into three loops wherein the three loops are a primary chilled water loop, a secondary chilled water loop, and an air loop, and an energy consumption of the primary loop is Ech, an energy consumption of the secondary loop is Epump, an energy consumption of the air loop is Efan;
train a plurality of deep Reinforcement learning (RL) agents using the generated simulation model, wherein the plurality of deep RL agents interacts with an environment of the building and learns to act using a Markov Decision Process (MDP) method;
deploy each of the plurality of trained deep RL agents in the one or more controllable electric equipment of the building, wherein
the primary chilled water loop is assigned with a first deep trained RL agent of the plurality of trained deep RL agents,
the secondary chilled water loop is assigned with a second deep trained RL agent of the plurality of trained deep RL agents, and
the air loop is assigned with a third deep trained RL agent of the plurality of trained deep RL agents,
each of the plurality of trained deep RL agents monitors one or more states affecting performance of the one or more controllable electric equipment in the building,
during an execution of each of the plurality of trained RL agents, a reward function is assigned to each of the plurality of trained RL agents,
the reward function comprises an energy component of the one or more controllable electrical equipment and a penalty, and
the reward function comprises an energy component,
the penalty includes a visual feedback of an associated controllable electrical equipment and degradation information of the associated controllable electrical equipment,
the reward function of the first deep trained RL agent is r1=Ech+a factor*the visual feedback,
the reward function of the second deep trained RL agent is r2=Epump+the factor*Ech, and
the reward function of the third deep trained RL agent is r3=Efan+a factor1*the visual feedback+a factor2*Ech;
trigger each of the plurality of trained deep RL agents to obtain a portion of the reward function associated with another deep RL agent, wherein
a first trained deep RL agent of the plurality of trained deep RL agents obtains a first specific portion of the portion of the reward function from a second trained deep RL agent of the plurality of trained deep RL agents,
the first specific portion of the reward function includes the energy component only, the penalty only or combination of both the energy component and the penalty,
the first specific portion of the reward function include the energy component and the penalty,
the first trained deep RL agent is associated with a first controllable electrical equipment of the one or more controllable electrical equipment,
the second trained deep RL agent is associated with a second controllable electrical equipment of the one or more controllable electrical equipment, and
information on the portion of the reward function is utilized for training the plurality of trained deep RL agents;
estimate, based on the obtained portion of the reward function, a global optimal control parameter list based on at least one optimal control parameter associated with each of the plurality of trained deep RL agents, wherein the optimal control parameter is learnt by each of the plurality of trained deep RL agents during an execution of the plurality of trained deep RL agents deployed in the one or more controllable electric equipment in the building, wherein the optimal control parameter comprises scheduling information of the associated controllable electrical equipment; and
fine tune a plurality of control parameters of the global optimal control parameter list to improve a performance of each of the plurality of trained deep RL agents.

### Claim 7 (depends on 6)

The system of claim 6, wherein the one or more controllable electrical equipment comprises one of one or more heating, ventilation, and air conditioning (HVAC) subsystems, one or more lighting equipment, computing loads systems or combinations thereof.

### Claim 8 (depends on 6)

The system of claim 6, wherein the one or more states comprise at least one of a temperature, a humidity, one or more ambient parameters, a lighting intensity, and an occupant density.

### Claim 9 (depends on 6)

The system of claim 6, wherein the penalty further comprises a thermal feedback.

### Claim 10 (depends on 6)

The system of claim 6, wherein the at least one optimal control parameter comprises at least one of a set point temperature, and a lighting intensity set point.

### Claim 11

One or more non-transitory machine-readable information storage mediums comprising one or more instructions which when executed by one or more hardware processors cause implementing multi-agent deep reinforcement learning for dynamically controlling electrical equipment in buildings by:
obtaining, via the one or more hardware processors, input data comprising (i) a design specification of one or more controllable electrical equipment installed and operating in the building and (ii) design details of the building associated thereof;
generating, via the one or more hardware processors, a simulation model using the input data, wherein the simulation model is built in offline by abstracting complex Heating, ventilation, and air conditioning (HVAC) loops of the building into three loops wherein the three loops are a primary chilled water loop, a secondary chilled water loop, and an air loop, and an energy consumption of the primary loop is Ech, an energy consumption of the secondary loop is Epump, an energy consumption of the air loop is Efan;
training, via the one or more hardware processors, a plurality of deep Reinforcement learning (RL) agents using the generated simulation model, wherein the plurality of deep RL agents interacts with an environment of the building and learns to act using a Markov Decision Process (MDP) method;
deploying, via the one or more hardware processors, each of the plurality of trained deep RL agents in the one or more controllable electric equipment of the building, wherein
the primary chilled water loop is assigned with a first deep trained RL agent of the plurality of trained deep RL agents,
the secondary chilled water loop is assigned with a second deep trained RL agent of the plurality of trained deep RL agents, and
the air loop is assigned with a third deep trained RL agent of the plurality of trained deep RL agents,
each of the plurality of trained deep RL agents monitors one or more states affecting performance of the one or more controllable electric equipment in the building,
during an execution of each of the plurality of trained RL agents, a reward function is assigned to each of the plurality of trained RL agents,
the reward function comprises an energy component of the one or more controllable electrical equipment and a penalty,
the reward function comprises an energy component, and
the penalty includes a visual feedback of an associated controllable electrical equipment and degradation information of the associated controllable electrical equipment, the reward function of the first deep trained RL agent is r1=Ech+a factor*the visual feedback,
the reward function of the second deep trained RL agent is r2=Epump+the factor*Ech, and
the reward function of the third deep trained RL agent is r3=Efan+a factor1*the visual feedback+a factor2*Ech;
triggering, via the one or more hardware processors, each of the plurality of trained deep RL agents, to obtain a portion of the reward function associated with another deep RL agent, wherein
a first trained deep RL agent of the plurality of trained deep RL agents obtains a first specific portion of the portion of the reward function from a second trained deep RL agent of the plurality of trained deep RL agents,
the first specific portion of the reward function includes the energy component only, the penalty only or combination of both the energy component and the penalty,
the first specific portion of the reward function include the energy component and the penalty,
the first trained deep RL agent is associated with a first controllable electrical equipment of the one or more controllable electrical equipment,
the second trained deep RL agent is associated with a second controllable electrical equipment of the one or more controllable electrical equipment, and
information on the portion of the reward function is utilized for training the plurality of trained deep RL agents;
estimating, via the one or more hardware processors, based on the obtained portion of the reward function, a global optimal control parameter list based on at least one optimal control parameter associated with each of the plurality of trained deep RL agents, wherein the at least one optimal control parameter is learnt by each of the plurality of trained deep RL agents during an execution of the plurality of trained deep RL agents deployed in the one or more controllable electric equipment in the building, wherein the optimal control parameter comprises scheduling information of the associated controllable electrical equipment; and
fine tuning a plurality of control parameters of the global optimal control parameter list to improve a performance of each of the plurality of trained deep RL agents.

### Claim 12 (depends on 11)

The one or more non-transitory machine-readable information storage mediums of claim 11, wherein the one or more controllable electrical equipment comprises one of one or more heating equipment, a ventilation system, and air conditioning (HVAC) subsystems, one or more lighting equipment, computing loads systems, or combinations thereof.

### Claim 13 (depends on 11)

The one or more non-transitory machine-readable information storage mediums of claim 11, wherein the one or more states comprise at least one of a temperature, a humidity, one or more ambient parameters, a lighting intensity, and an occupant density.

### Claim 14 (depends on 11)

The one or more non-transitory machine-readable information storage mediums of claim 11, wherein the penalty further comprises a thermal feedback.

### Claim 15 (depends on 11)

The one or more non-transitory machine-readable information storage mediums of claim 11, wherein the optimal control parameter comprises at least one of a set point temperature, and a lighting intensity set point.