Fault tolerance framework for networks of nodes

A system including instructions recorded on a non-transitory computer-readable storage medium and executable by at least one processor, the system comprising:

• the at least one processor;
• a middleware layer configured to cause the at least one processor to communicate with a plurality of networks of nodes and configured to cause the at least one processor to communicate with at least one back-end application, the middleware layer including
• a platform abstraction layer configured to:
• cause the at least one processor to receive, from the plurality of networks of nodes, first network-related data using a first communications protocol from a first network of nodes and second network-related data using a second communications protocol from a second network of nodes, wherein the first communications protocol is used by the nodes of the first network for communicating with one another within the first network and the second communications protocol is used by the nodes of the second network for communicating with one another within the second network, and
• provide the first network-related data and the second network-related data in a common protocol;
• and
• a fault management layer configured to:
• cause the at least one processor to construct and maintain a state model that stores events occurring in the first network of nodes and the second network of nodes, stores related state information, and stores network metadata for the first network of nodes and the second network of nodes,
• cause the at least one processor to receive the first network-related data and the second network-related data in the common protocol, and
• cause the at least one processor to determine a fault associated with an operation of the plurality of networks, based on the network-related data in the common protocol and on the state model,
• cause the at least one processor to determine a recovery method for responding to the fault, and
• cause the at least one processor to notify the platform abstraction layer of the recovery method, wherein the platform abstraction layer is further configured to:
• cause the at least one processor to determine a target node not associated with the fault within the first network of nodes when the recovery method includes deploying a service,
• cause the at least one processor to select a service injector from a plurality of service injectors, the selected service injector being compatible with the first communications protocol, and
• cause the at least one processor to deploy the service, using the selected service injector, to the target node for execution thereon to thereby recover the fault.

A method comprising:

• receiving network-related data associated with a plurality of networks of nodes at one of a plurality of message handlers, the plurality of message handlers each associated with a corresponding network of nodes and a corresponding communications protocol that is used by the corresponding network of nodes to conduct in-network communications within and among the corresponding nodes thereof;
• translating the network-related data from the corresponding communications protocol into a common communications protocol;
• providing the network-related data in the common communications protocol to a state model that stores state information related to the plurality of networks of nodes and network metadata for the plurality of networks of nodes;
• diagnosing a fault associated with an operation of the plurality of networks of nodes, based on the state model, the operation being associated with a source node of a first network of nodes within the plurality of networks;
• and
• recovering the fault by:
• determining a recovery method for recovering the operation despite the fault,
• determining a target node not associated with the fault within the first network of nodes when the recovery method includes deploying a service,
• selecting a service injector from a plurality of service injectors, the selected service injector being compatible with a communications protocol corresponding to the source node, and
• deploying, using the selected service injector, the service to the target node for execution on the target node to recover the fault.