Controlling operation of an electrical grid using reinforcement learning and multi-particle modeling

A computer-implemented method comprising:

• generating, by one or more computing systems, a model of a physical system whose operations include providing electrical power from multiple providers, wherein the model describes operational characteristics of the physical system and includes multiple rules that each has one or more conditions to evaluate and that specify restrictions involving a plurality of elements of the physical system and includes state information from sensors for the physical system and includes information about multiple control actions available to affect the providing of the electrical power and includes an indicated goal to maximize for the operations;
• and
• controlling, by the one or more computing systems and using the generated model, the operations of the physical system for an indicated time period, including:
• receiving, by the one or more computing systems, information for the indicated time period that includes projected electrical power production available from the multiple providers and includes projected electrical power load;
• generating, by the one or more computing systems, multiple particles that each represents a different set of state information for the physical system;
• propagating, by the one or more computing systems, and for each of a plurality of iterations during a time window that lasts for at least some of the indicated time period, the respective state information for each of the multiple particles to attempt to determine projected future state information for the particle that reflects an improved ability of the physical system to satisfy the projected electrical power load using the projected electrical power production available from the multiple providers in light of the indicated goal, including combining information for the multiple particles for use in a next iteration;
• determining, by the one or more computing systems, one or more of the multiple particles whose projected future state information provides a solution for the physical system to satisfy the projected electrical power load using the projected electrical power production available from the multiple providers in light of the indicated goal;
• and
• implementing, by the one or more computing systems, and using the solution for the determined one or more particles, at least one control action in the physical system to cause the multiple providers to satisfy the projected electrical power load for the indicated time period.

A system comprising:

• one or more hardware processors of one or more computing systems;
• and
• one or more memories with stored instructions that, when executed by at least one of the one or more hardware processors, cause the one or more computing systems to implement an automated control system for a physical system whose operations include providing electrical power, including:
• obtaining a model of the physical system describing operational characteristics of the physical system and including state information from sensors for the physical system and including additional information about multiple control actions available to affect the providing of the electrical power and including an indicated goal to maximize for the operations;
• and
• controlling, using the model, the operations of the physical system for an indicated time period, including:
• receiving information that includes projected electrical power production available from the physical system for the indicated time period and includes projected electrical power load for the indicated time period;
• generating multiple particles that each represents a different set of state information for the physical system;
• propagating, for each of a plurality of iterations during a time window that lasts for at least some of the indicated time period, the respective state information for each of the multiple particles to attempt to determine projected future state information for the particle that reflects an improved ability of the physical system to satisfy the projected electrical power load using the projected electrical power production in light of the indicated goal, including combining information for the multiple particles for use in a next iteration;
• determining, one or more of the multiple particles whose projected future state information provides a solution for the physical system to satisfy the projected electrical power load using the projected electrical power production in light of the indicated goal;
• and
• providing information about the solution for the determined one or more particles, to enable at least one control action in the physical system that is based on that solution to cause the physical system to satisfy the projected electrical power load for the indicated time period.

A non-transitory computer-readable medium having stored contents that cause one or more computing systems to perform automated operations including at least:

• generating, by the one or more computing systems and for a physical system, a model describing operational characteristics of the physical system and including multiple rules that each has one or more conditions to evaluate and that specify restrictions involving a plurality of elements of the physical system and including state information from sensors for the physical system and including information about multiple control actions available to affect operations of the physical system and includes an indicated goal to maximize for the operations;
• and
• controlling, by the one or more computing systems and using the generated model, the operations of the physical system for an indicated time period, including:
• receiving, by the one or more computing systems, information that includes projected functionality of an indicated type available from the physical system for the indicated time period and includes projected demand for the functionality for the indicated time period;
• generating, by the one or more computing systems, multiple particles that each represents a different set of state information for the physical system;
• determining, by the one or more computing systems and for each of a plurality of iterations during a time window that lasts for at least some of the indicated time period, and using the respective state information for each of the multiple particles, projected future state information for the particle that reflects an improved ability of the physical system to satisfy the projected demand for the functionality in light of the indicated goal, including combining information for the multiple particles for use in a next iteration;
• determining, by the one or more computing systems, one or more of the multiple particles whose projected future state information provides a solution for the physical system to satisfy the projected demand for the functionality in light of the indicated goal;
• and
• implementing, by the one or more computing systems, and using the solution for the determined one or more particles, at least one control action in the physical system to cause the physical system to satisfy the projected demand for the functionality for the indicated time period.