Failure analysis system for a distributed storage system

A non-transitory computer-readable medium having instructions stored thereon, which in response to execution by one or more processors, cause the one or more processors to perform or control performance of operations to address health issues indicative of operating conditions in a virtualized computing environment that includes at least one host, the operations comprising:

• monitoring, by a health check agent installed at the at least one host, health of the virtualized computing environment;
• detecting, based on health check information provided by the health check agent from the monitoring, a health issue that has manifested in the virtualized computing environment;
• generating, by an automated tool, a model that represents elements at multiple layers of the virtualized computing environment, and connections and relationships between the elements;
• using model-based reasoning to identify an element, amongst the elements, in the virtualized computing environment that is a source of the health issue,
• wherein the model-based reasoning uses the model, representing the connections and the relationships between the elements in the virtualized computing environment, to determine facts and rules for identification of the element that is the source of the health issue;
• using a decision tree analysis to identify a root cause of the health issue at the identified element, wherein a decision tree for use in the decision tree analysis is generated by injecting a fault into the virtualized computing environment to determine types, locations, and number of failures that are generated in the virtualized computing environment due to the injected fault;
• based on a result of the decision tree analysis that identifies the root cause of the health issue, generating a recommendation for remediation of the health issue;
• and
• performing the recommended remediation of the health issue.

A system to address health issues indicative of operating conditions in a virtualized computing environment that includes at least one host, the system comprising:

• a health check agent installed at the at least one host and configured to monitor health of the virtualized computing environment;
• a management server configured to detect, based on health check information provided by the health check agent, a health issue that has manifested in the virtualized computing environment;
• an automated tool configured to generate a model that represents elements at multiple layers of the virtualized computing environment, and connections and relationships between the elements;
• and
• a failure analysis system, coupled to the management server, configured to use model-based reasoning to identify an element, amongst the elements, in the virtualized computing environment that is a source of the health issue, wherein:
• the model-based reasoning uses the model, representing the connections and the relationships between the elements in the virtualized computing environment, to determine facts and rules for identification of the element that is the source of the health issue,
• the failure analysis system is further configured to use a decision tree analysis to identify a root cause of the health issue at the identified element, wherein a decision tree for use in the decision tree analysis is generated by the management server by injection of a fault into the virtualized computing environment to determine types, locations, and number of failures that are generated in the virtualized computing environment due to the injected fault,
• based on a result of the decision tree analysis that identifies the root cause of the health issue, the failure analysis system is configured to generate a recommendation for remediation of the health issue;
• and
• the management server is further configured to perform the recommended remediation of the health issue.