Systems and methods for multiple comparison correction during configurable application feature experiments

A non-transitory computer readable storage medium including instructions that, when executed by a processor, cause the processor to perform operations for configurable application feature experiments, comprising:

• receiving, by a first server computer system from configurable applications executed by a plurality of end user systems, event messages generated by the configurable applications after deployment of a feature treatment to the configurable applications, the event messages comprising metric values associated with a set of metrics;
• performing, by the first server computer system, a statistical analysis of a set of metric values collected from the configurable applications to determine whether the feature treatment caused a statistically significant change in values associated with corresponding one or more metrics indicative of the configurable applications, wherein the statistical analysis comprises adjusting one or more parameters used to perform the statistical analysis based on a multiple comparison correction analysis;
• and
• transmitting, by the first server computer system to a second server computer system, a message indicating a statistical impact of the feature treatment on each of the metrics in the set of metrics.

A first server computer system for configurable application feature experiments, comprising:

• a memory to store event messages;
• and
• a processing system coupled with the memory configured to:
• receive, from configurable applications executed by a plurality of end user systems, event messages generated by the configurable applications after deployment of a feature treatment to the configurable applications, the event messages comprising metric values associated with a set of metrics,
• perform a statistical analysis of a set of metric values collected from the configurable applications to determine whether the feature treatment caused a statistically significant change in values associated with corresponding one or more metrics indicative of the configurable applications, wherein the statistical analysis comprises adjusting one or more parameters used to perform the statistical analysis based on a multiple comparison correction analysis, and
• transmit, to a second server computer system, a message indicating a statistical impact of the feature treatment on each of the metrics in the set of metrics.