Automatic recovery engine with continuous recovery state machine and remote workflows

A non-transitory computer-readable medium having instructions stored thereon, instructions that when executed by one or more processors cause a machine to:

• receive or detect multiple states of one or more hosted services;
• automatically perform a root cause analysis to identify an underlying current problem within the one or more hosted services,
• wherein the root cause analysis is based, at least in part, on the multiple states of the one or more hosted services;
• and
• determine, based on the underlying current problem identified by the root cause analysis, a recovery workflow that includes a multiple step trouble shooting and execution of multiple workflows selected based on responses detected to executed workflows.

A method comprising:

• receiving, via a monitoring platform, a plurality of signals identifying conditions of one or more hosted services;
• tracking, via a finite state machine, states of the one or more hosted services that can be reached based on transition conditions met by the plurality of signals;
• identifying, at the monitoring platform, a root cause of a potential underlying problem consistent with the conditions of the one or more hosted services;
• determining, within a control plane, a recovery or mitigation workflow to resolve the potential underlying problem;
• and
• automatically executing, via the control plane, the recovery or mitigation workflow;
• wherein the recovery or mitigation workflow includes a multiple step trouble shooting and execution of multiple workflows automatically selected based on responses to executed workflows.