Systems and methods for real-time application anomaly detection and configuration

A method for application anomaly detection and remediation, comprising:

• receiving, by a server computer system, a set of parameters from a developer system, wherein the one or more parameters comprise the metric, a maximum length of time over which the metric should be monitored for degradations, a desired impact of the feature treatment on the metric, and an amount that the metric can deviate from the control value of the metric in a direction opposite a direction of the desired impact;
• receiving, by the server computer system, a plurality of event tracking messages generated by configurable applications executed by a first plurality of end user systems after a feature treatment is deployed to the configurable applications;
• associating, in a data store at the server computer system, the feature treatment with values of a metric from the event messages that are attributable to the feature treatment being executed by the one or more configurable applications, wherein the metric is a performance indicator associated with the execution of the configurable application;
• determining, by the server computer system, an impact of the feature treatment on the metric based on the values of the metric that are associated with the feature treatment comprising detecting when the feature treatment causes a statistically significant degradation of the metric based on the received set of parameters when compared to a control value of the metric determined from a second plurality of end user systems that are not exposed to the feature treatment by performing a one-tailed statistical analysis of the impact of the feature treatment on the metric using the values of the metric from the event tracking messages generated by the first plurality of end user systems, and the control value of the metric determined from the second plurality of end user systems, wherein the one-tailed statistical analysis is performed periodically with an increasing length of time between each successive one-tailed statistical analysis performed;
• and
• in response to detecting the statistically significant degradation of the metric, generating and transmitting one or more notifications to one or more users responsible for development of the configurable applications, deploying a different feature treatment that re-configures the execution of the configurable applications of end user systems that have the feature treatment, or a combination thereof.

A method for application anomaly detection and remediation, comprising:

• receiving, by a server computer system, a set of parameters from a developer system, wherein the one or more parameters comprise the metric, a maximum length of time over which the metric should be monitored for degradations, a desired impact of the feature treatment on the metric, and an amount that the metric can deviate from the control value of the metric in a direction opposite a direction of the desired impact;
• receiving a second set of parameters from the developer system, wherein the second set of parameters define a higher confidence level to be used in the one-tailed statistical analysis, a different amount that the metric can deviate from the control value of the metric, or a combination thereof;
• receiving, by the server computer system, a plurality of event tracking messages generated by configurable applications executed by a first plurality of end user systems after a feature treatment is deployed to the configurable applications;
• associating, in a data store at the server computer system, the feature treatment with values of a metric from the event messages that are attributable to the feature treatment being executed by the one or more configurable applications, wherein the metric is a performance indicator associated with the execution of the configurable application;
• determining, by the server computer system, an impact of the feature treatment on the metric based on the values of the metric that are associated with the feature treatment comprising detecting when the feature treatment causes a statistically significant degradation of the metric based on the received set of parameters or the received second set of parameters when compared to a control value of the metric determined from a second plurality of end user systems that are not exposed to the feature treatment by performing a one-tailed statistical analysis of the impact of the feature treatment on the metric using the values of the metric from the event tracking messages generated by the first plurality of end user systems, and the control value of the metric determined from the second plurality of end user systems, wherein the determining further comprises:
• in response to detecting when the feature treatment causes the statistically significant degradation of the metric based on the received set of parameters, generating and transmitting the one or more notifications to the one or more users responsible for development of the configurable applications, and
• in response to detecting when the feature treatment causes the statistically significant degradation of the metric based on the received set second of parameters, deploying the different feature treatment that re-configures the execution of the configurable applications of end user systems that have the feature treatment.

A non-transitory computer readable storage medium including instructions that, when executed by a processor, cause the processor to perform operations for application anomaly detection and remediation, the operations comprising:

• receiving, by a server computer system, a set of parameters from a developer system, wherein the one or more parameters comprise the metric, a maximum length of time over which the metric should be monitored for degradations, a desired impact of the feature treatment on the metric, and an amount that the metric can deviate from the control value of the metric in a direction opposite a direction of the desired impact, wherein receiving the set of parameters further comprises receiving a confidence level parameter from the developer system;
• receiving, by the server computer system, a plurality of event tracking messages generated by configurable applications executed by a first plurality of end user systems after a feature treatment is deployed to the configurable applications;
• associating, in a data store at the server computer system, the feature treatment with values of a metric from the event messages that are attributable to the feature treatment being executed by the one or more configurable applications, wherein the metric is a performance indicator associated with the execution of the configurable application;
• determining, by the server computer system, an impact of the feature treatment on the metric based on the values of the metric that are associated with the feature treatment comprising detecting when the feature treatment causes a statistically significant degradation of the metric based on the received set of parameters when compared to a control value of the metric determined from a second plurality of end user systems that are not exposed to the feature treatment comprising:
• performing a one-tailed statistical analysis of the impact of the feature treatment on the metric using the values of the metric from the event tracking messages generated by the first plurality of end user systems, and the control value of the metric determined from the second plurality of end user systems;
• and using the confidence level parameter to calculate a p-value threshold to be used in each one-tailed statistical analysis of the impact of the feature treatment on the metric which limits an accumulated error rate to a desired level as defined by the confidence level parameter;
• and
• in response to detecting the statistically significant degradation of the metric, generating and transmitting one or more notifications to one or more users responsible for development of the configurable applications, deploying a different feature treatment that re-configures the execution of the configurable applications of end user systems that have the feature treatment, or a combination thereof.

A system for application anomaly detection and remediation, comprising:

• a memory to store event tracking messages;
• and
• a processing system coupled with the memory configured to:
• receive a set of parameters from a developer system, wherein the one or more parameters comprise the metric, a maximum length of time over which the metric should be monitored for degradations, a desired impact of the feature treatment on the metric, and an amount that the metric can deviate from the control value of the metric in a direction opposite a direction of the desired impact, wherein the receipt of the set of parameters further comprises receipt of a confidence level parameter from the developer system,
• receive a plurality of event tracking messages generated by configurable applications executed by a first plurality of end user systems after a feature treatment is deployed to the configurable applications,
• associate, in the memory, the feature treatment with values of a metric from the event messages that are attributable to the feature treatment being executed by the one or more configurable applications, wherein the metric is a performance indicator associated with the execution of the configurable application,
• determine an impact of the feature treatment on the metric based on the values of the metric that are associated with the feature treatment comprising detecting when the feature treatment causes a statistically significant degradation of the metric based on the received set of parameters when compared to a control value of the metric determined from a second plurality of end user systems that are not exposed to the feature treatment comprising the processing system further configured to:
• perform a one-tailed statistical analysis of the impact of the feature treatment on the metric using the values of the metric from the event tracking messages generated by the first plurality of end user systems, and the control value of the metric determined from the second plurality of end user systems;
• and use the confidence level parameter to calculate a p-value threshold to be used in each one-tailed statistical analysis of the impact of the feature treatment on the metric which limits an accumulated error rate to a desired level as defined by the confidence level parameter, and
• in response to detecting the statistically significant degradation of the metric, generate and transmit one or more notifications to one or more users responsible for development of the configurable applications, deploy a different feature treatment that re-configures the execution of the configurable applications of end user systems that have the feature treatment, or a combination thereof.