System and method of monitoring and measuring cluster performance hosted by an iaas provider by means of outlier detection

A system for detecting outlier virtual resources that perform differently from peer virtual resources within a cluster of virtual resources that are expected to perform similarly, the virtual resources being provided by an independent Infrastructure-as-a-Service (IaaS) provider to an IaaS tenant for operating a widely distributed service, wherein the widely distributed service may be at least one of geographically dispersed, part of different communication networks, and disjoint, wherein the IaaS provider is responsible for selection of resources, wherein an operational capacity of the resources may change substantially and rapidly, and wherein the IaaS tenant has no direct control over, and limited visibility into, the selection of resources, the system comprising:

• a data gateway configured to collect substantially live system-level metrics related to the operation of the cluster of resources;
• and
• an analysis module configured to:
• compare the live system-level metrics for resources in the cluster of resources against live system-level metrics for other peer resources in the cluster of resources;
• and
• determine that a candidate resource is an outlier if a difference between the live system-level metrics for the candidate resource and the live system-level metrics for other peer resources in the cluster of resources exceeds a predetermined threshold, wherein the determination that the candidate resource is an outlier includes at least one of performing correlation analysis on the candidate resource, performing ANOVA analysis on the candidate resource, and performing regression analysis on the candidate resource.