Systems and methods for configurable application feature experiment alerting using sequential testing

A non-transitory computer readable storage medium including instructions that, when executed by a processor, cause the processor to perform operations for configurable application feature experiments, the operations comprising:

• receiving, by a computer server from an application developer system, data indicative of:
• a metric to be collected after a feature treatment is deployed to a plurality of configurable applications executed by a plurality of end user systems, wherein the metric is associated with a performance characteristic of the plurality of configurable applications;
• receiving, by the computer server from the configurable applications executed by the plurality of end user systems, feature treatment event messages generated by the configurable applications after deployment of the feature treatment, the event messages comprising metric values associated with the metric;
• performing, by the computer server, a sequential testing process using the metric values from the event messages to determine when statistical significance has been reached for the metric values satisfying or not satisfying a significance threshold associated with the metric;
• in response to detecting when the statistical significance has been reached based on results of generated by the sequential testing process, generating an alert message indicative of at least the metric, the significance threshold associated with the metric, and the determination of the statistical significance of the metric satisfying or not satisfying the significance threshold;
• and
• transmitting, by the computer server to the application developer system, the alert message.

A system for configurable application feature experiments, comprising:

• a memory to store feature treatment event messages;
• and
• a processing system coupled with the memory configured to:
• receive, from an application developer system, data indicative of:
• a metric to be collected after a feature treatment is deployed to a plurality of configurable applications executed by a plurality of end user systems, wherein the metric is associated with a performance characteristic of the plurality of configurable applications,
• receive, from the configurable applications executed by the plurality of end user systems, feature treatment event messages generated by the configurable applications after deployment of the feature treatment, the event messages comprising metric values associated with the metric,
• perform a sequential testing process using the metric values from the event messages to determine when statistical significance has been reached for the metric values satisfying or not satisfying a significance threshold associated with the metric,
• in response to detecting when the statistical significance has been reached based on results of generated by the sequential testing process, generate an alert message indicative of at least the metric, the significance threshold associated with the metric, and the determination of the statistical significance of the metric satisfying or not satisfying the significance threshold, and
• transmit, by the computer server to the application developer system, the alert message.