Multi-controller declarative fault management and coordination for microservices

A computer program product comprising a computer readable storage medium having program instructions embodied therewith, the program instructions executable by a computing device to cause the computing device to:

• implement, using at least a portion of multiple independent controllers, at least one registry comprising one or more policy rules associated with one or more actions to be carried out in connection with occurrence of one or more specified events;
• process information pertaining to at least one fault impacting multiple resources within a given system, wherein respective portions of the multiple resources are managed by the multiple independent controllers, wherein processing information pertaining to at least one fault comprises invoking, upon detecting occurrence of at least one of the one or more specified events, one or more callback functions of at least a portion of the multiple independent controllers, and wherein invoking the one or more callback functions comprise invoking at least one representational state transfer application programming interface endpoint associated with the one or more callback functions over at least one hypertext transfer protocol connection;
• determine, by each of at least a portion of the multiple independent controllers and based at least in part on the processing of the information, one or more desired resource states and multiple remediation actions;
• generate, based at least in part on one or more of the determined desired resource states and the multiple determined remediation actions, a sequential ordering of the multiple determined remediation actions to be carried out by the at least a portion of the multiple controllers, wherein generating the sequential ordering of the multiple determined remediation actions is based at least in part on system topology information associated with the given system, stack information associated with the given system, and system configuration information associated with the given system, wherein system topology information comprises information pertaining to connections between microservices associated with the given system;
• and
• automatically initiate execution of the multiple determined remediation actions in accordance with the generated sequential ordering.

A system comprising:

• a memory configured to store program instructions;
• and
• a processor operatively coupled to the memory to execute the program instructions to:
• implement, using at least a portion of multiple independent controllers, at least one registry comprising one or more policy rules associated with one or more actions to be carried out in connection with occurrence of one or more specified events;
• process information pertaining to at least one fault impacting multiple resources within a given system, wherein respective portions of the multiple resources are managed by the multiple independent controllers, wherein processing information pertaining to at least one fault comprises invoking, upon detecting occurrence of at least one of the one or more specified events, one or more callback functions of at least a portion of the multiple independent controllers, and wherein invoking the one or more callback functions comprise invoking at least one representational state transfer application programming interface endpoint associated with the one or more callback functions over at least one hypertext transfer protocol connection;
• determine, by each of at least a portion of the multiple independent controllers and based at least in part on the processing of the information, one or more desired resource states and multiple remediation actions;
• generate, based at least in part on one or more of the determined desired resource states and the multiple determined remediation actions, a sequential ordering of the multiple determined remediation actions to be carried out by the at least a portion of the multiple controllers, wherein generating the sequential ordering of the multiple determined remediation actions is based at least in part on system topology information associated with the given system, stack information associated with the given system, and system configuration information associated with the given system, wherein system topology information comprises information pertaining to connections between microservices associated with the given system;
• and
• automatically initiate execution of the multiple determined remediation actions in accordance with the generated sequential ordering.