# US 12,222,122 B2 · Optimized HVAC control using domain knowledge combined with deep reinforcement learning (DRL) > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,222,122 B2 - Document ID: US12222122B2 - Status: Granted - Kind: B2 - Application number: 17/935,420 - Filed: 2022-09-26 - Published: 2023-04-27 - Granted: 2025-02-11 - Earliest claimed: 2021-10-19 - Assignees: Tata Consultancy Services Limited - Applicants: Tata Consultancy Services Limited - Inventors: Sagar Kumar Verma; Supriya Agrawal; Venkatesh Ramanathan; Ulka Shrotri; Srinarayana Nagarathinam; Rajesh Jayaprakash; Aabriti Dutta - Web page: https://patentgrep.com/patent/US12222122B2 ## CPC - F24F 11/63: Electronic processing - G05B 15/02: electric - G05B 13/0265: {the criterion being a learning criterion} - F24F 2120/20: Feedback from users - G05B 2219/2614: HVAC, heating, ventillation, climate control - F24F: AIR-CONDITIONING; AIR-HUMIDIFICATION; VENTILATION; USE OF AIR CURRENTS FOR SCREENING (removing dirt or fumes from areas where they are produced B08B15/00; vertical ducts for carrying away waste gases from buildings E04F17/02; tops for chimneys or ventilating shafts, terminals for flues F23L17/02) - G05B: CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS (systems for controlling or regulating non-electric variables G05D; systems for regulating electric or magnetic variables G05F; control devices or systems insofar as characterised by mechanical features only G05G) ## Abstract HVAC control system's supervisory control is crucial for energy-efficient thermal comfort in buildings. The control logic is usually specified as ‘if-then-that-else’ rules that capture the domain expertise of HVAC operators, but they often have conflict that may lead to sub-optimal HVAC performance. Embodiments of the present disclosure provide a method and system for optimized Heating, ventilation, and air-conditioning (HVAC) control using domain knowledge combined with Deep Reinforcement Learning (DRL). The system disclosed utilizes Deep Reinforcement Learning (DRL) for conflict resolution in a HVAC control in combination with domain knowledge in form of control logic. The domain knowledge is predefined in an Expressive Decision Tables (EDT) engine via a formal requirement specifier consumable by the EDT engine to capture domain knowledge of a building for the HVAC control. ## Claims ### Claim 1 A processor implemented method for an optimized Heating, Ventilation, and Air-conditioning (HVAC) control of a building, the method comprising: receiving, via an Expressive Decision Tables (EDT) engine executed by one or more hardware processors, a plurality of HVAC parameters of the building, measured for a current time instance (t), wherein the plurality of HVAC parameters comprises: (i) a return air temperature (RAT), (ii) an occupancy count (OpCnt), (iii) an outside air temperature (OAT), (iv) an occupant discomfort measured in terms of predicted percentage dissatisfied (PPD) metric, (v) a HVAC energy consumption (EHVAC) and (vi) a current time, and wherein the occupant discomfort and the HVAC energy consumption (EHVAC) are measured with respect to a previous action item (at−1) triggered at a previous time instant (t−1); analyzing, by the EDT engine executed by the one or more hardware processors, the plurality of HVAC parameters in accordance with a rule set predefined for the HVAC control of the building to determine an action space (At) comprising more than one action items (at1 . . . atn) for the current time instance (t) corresponding to more than one rules that are satisfied from among the rule set, wherein the rule set is predefined in the EDT engine via a formal requirement specifier consumable by the EDT engine to capture domain knowledge of the building for the HVAC control, and wherein presence of more than one or more action items is indicative of presence of one or more conflicts in the domain knowledge; receiving, by a Deep Reinforcement Learning (DRL) agent executed by the one or more hardware processors, the action space (At), a current state (St) of the building from the EDT engine, and a current reward (Rt) received by the DRL agent for the previous action item (at−1), wherein the current state (St) is represented by a state tuple {OATt, RATt, OpCntt, τ} comprising one or more HVAC parameters from among the plurality of HVAC parameters, and τ representing time-of-day capturing time related variations in the one or more plurality of HVAC parameters; and selecting, by the DRL agent executed by the one or more hardware processors, an optimal control action item from among the action space (At) comprising the one or more action items (at1 . . . atn) that resolves the conflicts by maximizing a cumulative reward received over an episode, wherein a target cumulative reward is computed for current state action pair (St, at) providing an expected return over the episode starting from the current state St, following a policy, taking an action item at. ### Claim 2 (depends on 1) The method of claim 1, wherein the target cumulative reward comprises an immediate reward component and a component providing neural networks approximation of maximum cumulative reward that is expected from the next state that is S(t+1), wherein the immediate reward component is computed based on a consumption reward component (RC) and an occupant discomfort component (RE), wherein weightage of each component is defined by a tuning factor (α) and the immediate award for a successive time instant (t+1) is defined by Rt+1=αRC+(1−α) RE, and wherein the RC is based on ratio of the HVAC energy consumption (EHVAC) computed for the successive time instant to a rated maximum HVAC energy consumption as a result of taking action at to a rated maximum HVAC energy consumption, and the RE is based on the ration of the occupant discomfort for the successive time instant to a maximum allowed occupant discomfort as a result of taking action at to a maximum allowed occupant discomfort. ### Claim 3 (depends on 1) The method of claim 1, wherein the DRL agent is trained using a deep Q-Network (DQN) for selecting the optimal control action by applying an iterative process using episodic-epoch based learning over an truncated action space (A), a state (S) and the immediate reward corresponding to action item (a) taken from a previous state received from the EDT engine, wherein the DQN approximates an action-value function (Q-value) for the state (S) in each iteration returning the expected cumulative discounted reward over an episode starting from the state (S), following the policy, taking the action item a, and wherein the control action by providing preference to a highest Q-value based on a ε-greedy policy, and wherein the iterative process is applied over two DQNs further comprising a policy network Q and a target network Q to choose action and approximate the target cumulative reward respectively. ### Claim 4 (depends on 1) The method of claim 1, wherein one or more rules in the rule set include a time based component. ### Claim 5 A system for an optimized Heating, Ventilation, and Air-conditioning (HVAC) control of a building, the system comprising: a memory storing instructions; one or more Input/Output (I/O) interfaces; and one or more hardware processors coupled to the memory via the one or more I/O interfaces, wherein the one or more hardware processors are configured by the instructions to: receive, via an Expressive Decision Tables (EDT) engine executed by the one or more hardware processors a plurality of HVAC parameters of the building, measured for a current time instance (t), wherein the plurality of HVAC parameters comprising: (i) a return air temperature (RAT), (ii) an occupancy count (OpCnt), (iii) an outside air temperature (OAT), (iv) an occupant discomfort measured in terms of predicted percentage dissatisfied (PPD) metric, (v) a HVAC energy consumption (EHVAC) and (vi) a current time, wherein the occupant discomfort and the HVAC energy consumption (EHVAC) are measured with respect to a previous action item (at−1) triggered at a previous time instant (t−1); analyze, by the EDT engine, the plurality of HVAC parameters in accordance with a rule set predefined for the HVAC control of the building to determine an action space (At) comprising more than one action items (at1 . . . atn) for the current time instance (t) corresponding to more than one rules that are satisfied from among the rule set, wherein the rule set is predefined in the EDT engine via a formal requirement specifier consumable by the EDT engine to capture domain knowledge of the building for the HVAC control, and wherein presence of more than one or more action items is indicative of presence of one or more conflicts in the domain knowledge; receive, by a Deep Reinforcement Learning (DRL) agent executed by the one or more hardware processors, the action space (At), a current state (St) of the building from the EDT engine, and a current reward (Rt) received by the DRL agent for the previous action item (at−1), wherein the current state (St) is represented by a state tuple {OATt, RATt, OpCntt, τ} comprising one or more HVAC parameters from among the plurality of HVAC parameters, and τ representing time-of-day capturing time related variations in the one or more plurality of HVAC parameters; and select, by the DRL agent, an optimal control action item from among the action space (At) comprising the one or more action items (at1 . . . atn) that resolves the conflicts by maximizing a cumulative reward received over an episode, wherein a target cumulative reward is computed for current state action pair (St, at) providing an expected return over the episode starting from the current state St, following a policy, taking an action item at. ### Claim 6 (depends on 5) The system of claim 5, wherein the target cumulative reward comprises an immediate reward component and a component providing neural networks approximation of maximum cumulative reward that is expected from the next state that is S(t+1), wherein the immediate reward component is computed based on a consumption reward component (RC) and an occupant discomfort component (RE), wherein weightage of each component is defined by a tuning factor (α) and the immediate award for a successive time instant (t+1) is defined by Rt+1=αRC+(1−α) RE, and wherein the RC is based on ratio of the HVAC energy consumption (EHVAC) computed for the successive time instant to a rated maximum HVAC energy consumption as a result of taking action at to a rated maximum HVAC energy consumption, and the RE is based on the ration of the occupant discomfort for the successive time instant to a maximum allowed occupant discomfort as a result of taking action at to a maximum allowed occupant discomfort. ### Claim 7 (depends on 5) The system of claim 5, wherein the DRL agent is trained using a deep Q-Network (DQN) for selecting the optimal control action by applying an iterative process using episodic-epoch based learning over an truncated action space (A), a state (S) and the immediate reward corresponding to action item (a) taken from a previous state received from the EDT engine, wherein the DQN approximates an action-value function (Q-value) for the state (S) in each iteration returning the expected cumulative discounted reward over an episode starting from the state (S), following the policy, taking the action item a, and wherein the control action by providing preference to a highest Q-value based on a ε-greedy policy, and wherein the iterative process is applied over two DQNs comprising a policy network Q and a target network Q to choose action and approximate the target cumulative reward respectively. ### Claim 8 (depends on 5) The system of claim 5, wherein one or more rules in the rule set include a time based component. ### Claim 9 One or more non-transitory machine-readable information storage mediums comprising one or more instructions which when executed by one or more hardware processors cause: receiving, via an Expressive Decision Tables (EDT) engine executed by one or more hardware processors, a plurality of HVAC parameters of the building, measured for a current time instance (t), wherein the plurality of HVAC parameters comprising: (i) a return air temperature (RAT), (ii) an occupancy count (OpCnt), (iii) an outside air temperature (OAT), (iv) an occupant discomfort measured in terms of predicted percentage dissatisfied (PPD) metric, (v) a HVAC energy consumption (EHVAC) and (vi) a current time, wherein the occupant discomfort and the HVAC energy consumption (EHVAC) are measured with respect to a previous action item (at−1) triggered at a previous time instant (t−1); analyzing, by the EDT engine executed by the one or more hardware processors, the plurality of HVAC parameters in accordance with a rule set predefined for the HVAC control of the building to determine an action space (At) comprising more than one action items (at1 . . . atn) for the current time instance (t) corresponding to more than one rules that are satisfied from among the rule set, wherein the rule set is predefined in the EDT engine via a formal requirement specifier consumable by the EDT engine to capture domain knowledge of the building for the HVAC control, and wherein presence of more than one or more action items is indicative of presence of one or more conflicts in the domain knowledge; receiving, by a Deep Reinforcement Learning (DRL) agent executed by the one or more hardware processors, the action space (At), a current state (St) of the building from the EDT engine, and a current reward (Rt) received by the DRL agent for the previous action item (at−1), wherein the current state (St) is represented by a state tuple {OATt, RATt, OpCntt, τ} comprising one or more HVAC parameters from among the plurality of HVAC parameters, and τ representing time-of-day capturing time related variations in the one or more plurality of HVAC parameters; and selecting, by the DRL agent executed by the one or more hardware processors, an optimal control action item from among the action space (At) comprising the one or more action items (at1 . . . atn) that resolves the conflicts by maximizing a cumulative reward received over an episode, wherein a target cumulative reward is computed for current state action pair (St, at) providing an expected return over the episode starting from the current state St, following a policy, taking an action item at. ### Claim 10 (depends on 9) The one or more non-transitory machine-readable information storage mediums of claim 9, wherein the target cumulative reward comprises an immediate reward component and a component providing neural networks approximation of maximum cumulative reward that is expected from the next state that is S(t+1), wherein the immediate reward component is computed based on a consumption reward component (RC) and an occupant discomfort component (RE), wherein weightage of each component is defined by a tuning factor (α) and the immediate award for a successive time instant (t+1) is defined by Rt+1=αRC+(1−α) RE, and wherein the RC is based on ratio of the HVAC energy consumption (EHVAC) computed for the successive time instant to a rated maximum HVAC energy consumption as a result of taking action at to a rated maximum HVAC energy consumption, and the RE is based on the ration of the occupant discomfort for the successive time instant to a maximum allowed occupant discomfort as a result of taking action at to a maximum allowed occupant discomfort. ### Claim 11 (depends on 9) The one or more non-transitory machine-readable information storage mediums of claim 9, wherein the DRL agent is trained using a deep Q-Network (DQN) for selecting the optimal control action by applying an iterative process using episodic-epoch based learning over an truncated action space (A), a state (S) and the immediate reward corresponding to action item (a) taken from a previous state received from the EDT engine, wherein the DQN approximates an action-value function (Q-value) for the state (S) in each iteration returning the expected cumulative discounted reward over an episode starting from the state (S), following the policy, taking the action item a, and wherein the control action by providing preference to a highest Q-value based on a ε-greedy policy, and wherein the iterative process is applied over two DQNs comprising a policy network Q and a target network Q to choose action and approximate the target cumulative reward respectively. ### Claim 12 (depends on 9) The one or more non-transitory machine-readable information storage mediums of claim 9, wherein one or more rules in the rule set include a time based component.