# US 12,760,920 B2 · Real-time carbon footprint reduction controller > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,760,920 B2 - Document ID: US12760920B2 - Status: Granted - Kind: B2 - Application number: 18/492,176 - Filed: 2023-10-23 - Published: 2024-11-28 - Granted: 2026-10-13 - Earliest claimed: 2023-05-26 - Assignees: Hewlett Packard Enterprise Development LP - Applicants: Hewlett Packard Enterprise Development LP - Inventors: Soumyendu Sarkar; Avisek Naug; Ricardo Luna Gutierrez; Antonio Guillen Perez; Sahand Ghorbanpour - Web page: https://patentgrep.com/patent/US12760920B2 ## CPC - F24F 11/46: Improving electric energy efficiency or saving - F24F 2140/50: Load - F24F 2140/60: Energy consumption - F24F: AIR-CONDITIONING; AIR-HUMIDIFICATION; VENTILATION; USE OF AIR CURRENTS FOR SCREENING (removing dirt or fumes from areas where they are produced B08B15/00; vertical ducts for carrying away waste gases from buildings E04F17/02; tops for chimneys or ventilating shafts, terminals for flues F23L17/02) ## Abstract Systems and methods are provided for optimizing energy consumption, flexible load shifting, and battery operation decisions simultaneously in real-time through Reinforcement Learning (RL). Examples include obtaining states of a system comprising a plurality of subsystems and receiving, by RL agents, rewards from a digital twin of the system, the rewards comprising a plurality of rewards each associated with a subsystem. The example also include determining actions, by the RL agents, based on the states and each of the rewards. Each RL agent is associated with a subsystem and assigns a weight to a reward corresponding to the associated subsystem that is greater than weights assigned to rewards of the other subsystems. The system can then be controlled according to the actions to transition the system to updated states. ## Claims ### Claim 1 A method, comprising: obtaining state data representative of states of the system, the system comprising a plurality of subsystems, wherein the plurality of subsystems comprises a cooling subsystem, a load shifting subsystem, and an energy storage subsystem; receiving, by a plurality of reinforcement learning agents, reward data from a digital twin of the system that simulates operations of the system, the reward data comprising a plurality of rewards each associated with a subsystem of the plurality of subsystems, and wherein: a first reward of the plurality of rewards corresponding to the cooling subsystem is based on a measure of energy consumed by the system and energy costs, a second reward of the plurality of rewards corresponding to the load shifting subsystem is based on a carbon footprint of the load shifting subsystem and unallocated workload, and a third reward of the plurality of rewards corresponding to the energy storage subsystem is based on a carbon footprint of the energy storage subsystem; determining a plurality of actions, by the plurality of reinforcement learning agents, based on the state data and each of the plurality of rewards, wherein each reinforcement learning agent of the plurality of reinforcement learning agents is associated with a subsystem of the plurality of subsystems and assigns a weight to a reward of the plurality of rewards corresponding to an associated subsystem of the plurality of subsystems that is greater than weights assigned to rewards corresponding to other subsystems of the plurality of subsystems; and transition the system to updated states according to the plurality of actions. ### Claim 2 (depends on 1) The method of claim 1, wherein the system is a data center. ### Claim 3 (depends on 1) The method of claim 1, wherein an action determined by a reinforcement learning agent associated with the cooling subsystem is dependent on an action determined by a reinforcement learning agent associated with the load shifting subsystem and an action determined by a reinforcement learning agent associated with the energy storage subsystem. ### Claim 4 (depends on 1) The method of claim 1, wherein an action determined by a reinforcement learning agent associated with the cooling subsystem is a cooling setpoint for the system, where an action determined by a reinforcement learning agent associated with the load shifting subsystem is an allocation of workload, and wherein an action determined by a reinforcement learning agent associated with the energy storage subsystem is a charge or discharge amount. ### Claim 5 (depends on 1) The method of claim 1, wherein an action determined by a reinforcement learning agent associated with the load shifting subsystem is dependent on an action determined by a reinforcement learning agent associated with the cooling subsystem and an action determined by a reinforcement learning agent associated with the energy storage subsystem. ### Claim 6 (depends on 1) The method of claim 1, wherein an action determined by a reinforcement learning agent associated with the energy storage subsystem is dependent on an action determined by a reinforcement learning agent associated with the cooling subsystem and an action determined by a reinforcement learning agent associated with the load shifting subsystem. ### Claim 7 (depends on 1) The method of claim 1, wherein the state data comprises internal states of the system and environmental states of an environment for the system, wherein the internal states comprise one or more of allocated workload, unallocated workload, temperature of the system, energy consumption by the system, or energy stored by the system, and wherein the environmental states comprise one or more of grid carbon intensity, weather, or time. ### Claim 8 A system for carbon footprint reduction of a system, comprising: a memory storing instructions; and one or more processors coupled to the memory and configured to execute the instructions to: obtain state data representative of states of the system, wherein the state data comprises internal states of the system and environmental states of an environment for the system, wherein the internal states comprise one or more of allocated workload, unal located workload, temperature of the system, energy consumption by the system, and energy stored by the system, and wherein the environmental states comprise one or more of grid carbon intensity, weather, and time, and wherein the system comprising a plurality of subsystems; receive reward data from a digital twin of the system that simulates operations of the system, the reward data comprising a plurality of rewards each associated with a subsystem of the plurality of subsystems; determine a plurality of actions, by a plurality of reinforcement learning agents, based on the state data and each of the plurality of rewards, wherein each reinforcement learning agent of the plurality of reinforcement learning agents is associated with a subsystem of the plurality of subsystems and assigns a weight to a reward of the plurality of rewards corresponding to the an associated subsystem of the plurality of subsystems that is greater than weights assigned to rewards corresponding to the other subsystems of the plurality of subsystems; and transition the system to updated states according to the plurality of actions. ### Claim 9 (depends on 8) The system of claim 8, wherein the system is a data center, and wherein the plurality of subsystems comprises a cooling subsystem, a load shifting subsystem, and an energy storage subsystem. ### Claim 10 (depends on 9) The system of claim 9, wherein an action determined by a reinforcement learning agent associated with the cooling subsystem is dependent on an action determined by a reinforcement learning agent associated with the load shifting subsystem and an action determined by a reinforcement learning agent associated with the energy storage subsystem. ### Claim 11 (depends on 9) The system of claim 9, wherein an action determined by a reinforcement learning agent associated with the cooling subsystem is a cooling setpoint for the system, where an action determined by a reinforcement learning agent associated with the load shifting subsystem is an allocation of workload, and wherein an action determined by a reinforcement learning agent associated with the energy storage subsystem is a charge or discharge amount. ### Claim 12 (depends on 9) The system of claim 9, wherein an action determined by a reinforcement learning agent associated with the load shifting subsystem is dependent on an action determined by a reinforcement learning agent associated with the cooling subsystem and an action determined by a reinforcement learning agent associated with the energy storage subsystem. ### Claim 13 (depends on 9) The system of claim 9, wherein an action determined by a reinforcement learning agent associated with the energy storage subsystem is dependent on an action determined by a reinforcement learning agent associated with the cooling subsystem and an action determined by a reinforcement learning agent associated with the load shifting subsystem. ### Claim 14 A system, comprising: a hardware processor; a non-transitory machine-readable storage medium storing instructions that, when executed by the processor, cause the processor to: initiate a digital twin of a data center, the digital twin comprising a plurality of subsystem models corresponding to a plurality of subsystems of the data center and configured to simulate states of received environmental states of an environment for the data center, comprising one or more of grid carbon intensity, weather, or and time, for each subsystem model and determine rewards for each subsystem model, wherein the simulated states comprise one or more of allocated workload, unallocated workload, temperature of the data center, energy consumption by the data center, and energy stored by the data center; and initiate a plurality of reinforcement learning agents interfaced with the digital twin and configured to: receive the simulated states and the rewards from a digital twin, aggregate the rewards, determine actions for each subsystem model based on the aggregated rewards, and generate control signals for each subsystem according to the determined actions, wherein the data center is configured based on the control signals. ### Claim 15 (depends on 14) The system of claim 14, wherein the plurality of subsystem models comprises a cooling subsystem model, a load shifting subsystem model, and an energy storage subsystem model. ### Claim 16 (depends on 14) The system of claim 14, wherein each reinforcement learning agent of the plurality of reinforcement learning agents is associated with a subsystem model of the plurality of subsystem models, and wherein aggregating the rewards comprises: by each reinforcement learning agent of the plurality of reinforcement learning agents, assigning a weight to a reward of the rewards determined by an associated subsystem model that is greater than weights assigned to rewards determined by the other subsystem models.