# US 12,217,153 B2 · Training server and method for generating a predictive model of a neural network through distributed reinforcement learning > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,217,153 B2 - Document ID: US12217153B2 - Status: Granted - Kind: B2 - Application number: 18/511,309 - Filed: 2023-11-16 - Published: 2024-03-14 - Granted: 2025-02-04 - Earliest claimed: 2019-08-26 - Assignees: Distech Controls Inc. - Applicants: Distech Controls Inc. - Inventors: Steve Lupien; Francois Gervais - Web page: https://patentgrep.com/patent/US12217153B2 ## CPC - G06N 3/047: Probabilistic or stochastic networks - G06N 3/0499: Feedforward networks - G06N 3/063: using electronic means - G06N 3/08: Learning methods - G06N 3/092: Reinforcement learning - G06N 3/098: Distributed learning, e.g. federated learning - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS ## Abstract Interactions between a training server and a plurality of environment controllers are used for updating the weights of a predictive model used by a neural network executed by the plurality of environment controllers. Each environment controller executes the neural network using a current version of the predictive model to generate outputs based on inputs, modifies the outputs, and generates metrics representative of the effectiveness of the modified outputs for controlling the environment. The training server collects the inputs, the corresponding modified outputs, and the corresponding metrics from the plurality of environment controllers. The collected inputs, modified outputs and metrics are used by the training server for updating the weights of the current predictive model through reinforcement learning. A new predictive model comprising the updated weights is transmitted to the environment controllers to be used in place of the current predictive model. ## Claims ### Claim 1 An environment controller comprising: at least one communication interface; memory for storing a predictive model comprising weights of a neural network; and a processing unit comprising one or more processor configured to: (a) determine at least one environmental characteristic value in an area; (b) receive at least one set point via one of the at least one communication interface or a user interface of the environment controller; (c) execute a neural network inference engine using the predictive model for generating one or more output based on inputs, the one or more output comprising one or more command for controlling a controlled appliance, the inputs comprising the at least one environmental characteristic value in the area and the at least one set point; (d) modify the one or more command; (e) transmit the one or more modified command to the controlled appliance via the at least one communication interface; (f) generate at least one metric representative of an execution of the one or more modified command by the controlled appliance; (g) transmit the inputs, the one or more output and the at least one metric to a training server via the at least one communication interface, the one or more output comprising the one or more modified command; and (h) receive an update of the predictive model comprising updated weights from the training server via the at least one communication interface. ### Claim 2 (depends on 1) The environment controller of claim 1, wherein steps (a) to (g) are repeated a plurality of times before the occurrence of step (h). ### Claim 3 (depends on 1) The environment controller of claim 1, wherein modifying the one or more command comprises at least one of the following: applying a random modification to the one or more command, applying a modification selected among a set of one or more pre-defined modification to the one or more command, and applying a modification based on configuration data included in a configuration message received via the at least one communication interface. ### Claim 4 (depends on 1) The environment controller of claim 1, wherein the at least one metric comprises at least one of the following: at least one updated environmental characteristic value in the area determined by the processing unit after the transmission of the one or more modified command to the controlled appliance, and at least one measurement by the processing unit of a time required for reaching at least one corresponding environmental state in the area after the transmission of the one or more modified command to the controlled appliance. ### Claim 5 (depends on 1) The environment controller of claim 1, wherein the at least one environmental characteristic value in the area comprises at least one of the following: a current temperature in the area, a current humidity level in the area, a current carbon dioxide (CO2) level in the area, and a current occupancy of the area. ### Claim 6 (depends on 1) The environment controller of claim 1, wherein the at least one environmental characteristic value in the area comprises at least one of the following: a plurality of consecutive temperature measurements in the area, a plurality of consecutive humidity level measurements in the area, a plurality of consecutive carbon dioxide (CO2) level measurements in the area, and a plurality of consecutive determinations of an occupancy of the area. ### Claim 7 (depends on 1) The environment controller of claim 1, wherein determining the environmental characteristic value in the area consists in receiving the environmental characteristic value from a sensor via the at least one communication interface. ### Claim 8 (depends on 1) The environment controller of claim 1, wherein the at least one set point comprises at least one of the following: a target temperature, a target humidity level, and a target CO2 level. ### Claim 9 (depends on 1) The environment controller of claim 1, wherein the one or more processor is further configured to determine at least one characteristic of the area, the inputs further including the at least one characteristic of the area; wherein the at least one characteristic of the area comprises at least one of the following: an area type identifier selected among a plurality of area type identifiers, one or more geometric characteristics of the area, and a human activity in the area. ### Claim 10 (depends on 1) The environment controller of claim 1, wherein the controlled appliance consists of a heating, ventilating and air-conditioning (HVAC) appliance. ### Claim 11 (depends on 1) The environment controller of claim 1, wherein the one or more command includes at least one of the following: a command for controlling a speed of a fan, a command for controlling a pressure generated by a compressor, and a command for controlling a rate of an airflow through a valve. ### Claim 12 A method for improving a predictive model of a neural network used for performing environment control, the method comprising: storing in a memory of a computing device a predictive model comprising weights of a neural network; (a) determining by a processing unit of the computing device at least one environmental characteristic value in an area; (b) receiving at least one set point via one of a communication interface of the computing device or a user interface of the computing device; (c) executing by the processing unit of the computing device a neural network inference engine using the predictive model for generating one or more output based on inputs, the one or more output comprising one or more command for controlling a controlled appliance, the inputs comprising the at least one environmental characteristic value in the area and the at least one set point; (d) modifying by the processing unit of the computing device the one or more command; (e) transmitting the one or more modified command to the controlled appliance via the communication interface of the computing device; (f) generating by the processing unit of the computing device at least one metric representative of an execution of the one or more modified command by the controlled appliance; (g) transmitting the inputs, the one or more output and the at least one metric to a training server via the communication interface, the one or more output comprising the one or more modified command; and (h) receiving an update of the predictive model comprising updated weights from the training server via the communication interface. ### Claim 13 (depends on 12) The method of claim 12, wherein steps (a) to (g) are repeated a plurality of times before the occurrence of step (h). ### Claim 14 (depends on 12) The method of claim 12, wherein modifying the one or more command comprises at least one of the following: applying a random modification to the one or more command, applying a modification selected among a set of one or more pre-defined modification to the one or more command, and applying a modification based on configuration data included in a configuration message received via the communication interface. ### Claim 15 (depends on 12) The method of claim 12, wherein the at least one metric comprises at least one of the following: at least one updated environmental characteristic value in the area determined by the processing unit after the transmission of the one or more modified command to the controlled appliance, and at least one measurement by the processing unit of the computing device of a time required for reaching at least one corresponding environmental state in the area after the transmission of the one or more modified command to the controlled appliance. ### Claim 16 (depends on 12) The method of claim 12, wherein the at least one environmental characteristic value in the area comprises at least one of the following: a current temperature in the area, a current humidity level in the area, a current carbon dioxide (CO2) level in the area, and a current occupancy of the area. ### Claim 17 (depends on 12) The method of claim 12, wherein the at least one environmental characteristic value in the area comprises at least one of the following: a plurality of consecutive temperature measurements in the area, a plurality of consecutive humidity level measurements in the area, a plurality of consecutive carbon dioxide (CO2) level measurements in the area, and a plurality of consecutive determinations of an occupancy of the area. ### Claim 18 (depends on 12) The method of claim 12, wherein the at least one set point comprises at least one of the following: a target temperature, a target humidity level, and a target CO2 level. ### Claim 19 (depends on 12) The method of claim 12, further comprising determining by the processing unit of the computing device at least one characteristic of the area, the inputs further including the at least one characteristic of the area; wherein the at least one characteristic of the area comprises at least one of the following: an area type identifier selected among a plurality of area type identifiers, one or more geometric characteristics of the area, and a human activity in the area. ### Claim 20 A non-transitory computer program product comprising instructions executable by a processing unit of a computing device, the execution of the instructions by the processing unit of the computing device providing for improving a predictive model of a neural network used for performing environment control by: storing in a memory of the computing device a predictive model comprising weights of a neural network; (a) determining at least one environmental characteristic value in an area; (b) receiving at least one set point via one of a communication interface of the computing device or a user interface of the computing device; (c) executing a neural network inference engine using the predictive model for generating one or more output based on inputs, the one or more output comprising one or more command for controlling a controlled appliance, the inputs comprising the at least one environmental characteristic value in the area and the at least one set point; (d) modifying the one or more command; (e) transmitting the one or more modified command to the controlled appliance via the communication interface of the computing device; (f) generating at least one metric representative of an execution of the one or more modified command by the controlled appliance; (g) transmitting the inputs, the one or more output and the at least one metric to a training server via the communication interface, the one or more output comprising the one or more modified command; and (h) receiving an update of the predictive model comprising updated weights from the training server via the communication interface.