Learning control system and learning control method

A learning control system controlling multi-agent learning in a predetermined system, the learning control system comprising:

• a computer system including a plurality of computers which are programmed to execute:
• a plurality of agents that are connected to a plurality of subsystems of the predetermined system, respectively, and perform learning to control the plurality of subsystems, which are control targets, using a plurality of control models;
• and
• a learning management agent that is connected to the plurality of agents for communication and manages and controls learning of each of the plurality of agents,
• wherein each of the plurality of agents:
• receives information including a respective control model of the plurality of control models from the learning management agent,
• calculates an evaluation value of a respective subsystem of the plurality of subsystems based on a state value of the respective subsystem,
• inputs the state value, determines a behavior value of the respective subsystem through calculation of the respective control model, and outputs the behavior value to the respective subsystem,
• updates a parameter of the respective control model in accordance with learning, and
• transmits information including the respective control model and the evaluation value of the respective subsystem to the learning management agent, and
• wherein the learning management agent:
• constructs a plurality of experiment systems including a plurality of control model sets in the plurality of agents in a state in which the plurality of agents are connected to the plurality of subsystems and controls learning in a plurality of generations in the plurality of experiment systems, and
• evaluates the plurality of experiment systems of a current generation of the plurality of generations based on the evaluation value of each of the plurality of subsystems, determines a plurality of updating control model sets in the plurality of experiment system of a next generation of the plurality of generations based on an evaluation result of the plurality of experiment systems of the current generation, and transmits information regarding the updating control model sets to the corresponding plurality of agents,
• wherein each of the plurality of agents includes:
• a first communication unit that communicates with the respective subsystem which is the control target thereof,
• a second communication unit that communicates with the learning management agent,
• a state acquisition unit that acquires information including the state value from the respective subsystem,
• an evaluation unit that calculates the evaluation value of the respective subsystem based on the state value,
• a control unit that calculates a control value and a behavior value of the respective subsystem based on the state value and the respective control model,
• a learning unit that stores the state value, the control value, and the behavior value and updates the parameter of the respective control model in accordance with the learning, and
• a behavior output unit that calculates the behavior value of the respective subsystem from the control value and outputs the behavior value to the respective subsystem, and
• wherein the learning management agent includes:
• a communication unit that communicates with the plurality of agents,
• an experiment system evaluation unit that manages the plurality of experiment systems, evaluates the plurality of experiment systems of the current generation based on the evaluation value of each of the plurality of subsystems, and outputs the evaluation result, and
• a control model operation unit that determines the plurality of updating control model sets in the plurality of experiment systems of the next generation based on the evaluation result and outputs the information regarding the updating control model sets to the corresponding plurality of agents.

A learning control system controlling multi-agent learning in a predetermined system, the learning control system comprising:

• a computer system including a plurality of computers which are programmed to execute:
• a plurality of agents that are connected to a plurality of subsystems of the predetermined system, respectively, and perform learning to control the plurality of subsystems, which are control targets, using a plurality of control models;
• and
• a learning management agent that is connected to the plurality of agents for communication and manages and controls learning of each of the plurality of agents,
• wherein each of the plurality of agents:
• receives information including a respective control model of the plurality of control models from the learning management agent,
• calculates an evaluation value of a respective subsystem of the plurality of subsystems based on a state value of the respective subsystem,
• inputs the state value, determines a behavior value of the respective subsystem through calculation of the respective control model, and outputs the behavior value to the respective subsystem,
• updates a parameter of the respective control model in accordance with learning, and
• transmits information including the respective control model and the evaluation value of the respective subsystem to the learning management agent, and
• wherein the learning management agent:
• constructs a plurality of experiment systems including a plurality of control model sets in the plurality of agents in a state in which the plurality of agents are connected to the plurality of subsystems and controls learning in a plurality of generations in the plurality of experiment systems, and
• evaluates the plurality of experiment systems of a current generation of the plurality of generations based on the evaluation value of each of the plurality of subsystems, determines a plurality of updating control model sets in the plurality of experiment system of a next generation of the plurality of generations based on an evaluation result of the plurality of experiment systems of the current generation, and transmits information regarding the updating control model sets to the corresponding plurality of agents,
• wherein the learning management agent:
• performs an evolution strategy calculation process on the plurality of control model sets in the plurality of experiment systems of the current generation to generate a plurality of different control model sets, and
• selects one control model of the plurality of control models by comparing the plurality of control model sets of the current generation to the plurality of generated different control model sets for each respective control model of the plurality of agents and determines the plurality of updating control model sets in the plurality of experiment systems of the next generation.