Sample ratio mismatch diagnosis tool

A method for determining and presenting an indication of a sample ratio mismatch (SRM) for a scorecard of an online controlled experiment (OCE), the method comprising:

• receiving a first scorecard from the OCE, wherein the OCE includes a treatment experience, a control experience, and an expected treatment-to-control (T/C) ratio, and wherein the first scorecard is for a first period of time;
• determining an observed first T/C ratio for the first scorecard;
• determining a first confidence interval using the observed first T/C ratio based on the expected T/C ratio and a threshold, the first confidence interval determined using a difference between the observed first T/C ratio and the expected T/C ratio;
• displaying, on a graphical user interface of a display screen, a marker representing the observed first T/C ratio;
• displaying, on the graphical user interface of the display screen, a line representing the first confidence interval;
• displaying, on the graphical user interface of the display screen, a line representing the expected T/C ratio, wherein whether the line representing the first confidence interval intersects with the line representing the expected T/C ratio indicates whether the first scorecard has an SRM;
• determining, based on the line representing the first confidence interval not intersecting the line representing the expected T/C ratio, that the SRM exists in the first scorecard;
• and
• before completing a second scorecard of the OCE, the second scorecard more expensive to compute than the first scorecard, stopping the OCE based on whether the first scorecard has an SRM.

A method comprising:

• detecting a sample ratio mismatch (SRM) in two online controlled experiments (OCEs) for a product;
• determining that the two OCEs use a same assignment mechanism and data pipeline;
• analyzing a first segment and a second segment included in the two OCEs for the product to determine segment SRMs, the first segment representing a first set of participants sharing a first attribute value and the second segment representing a second set of participants sharing a second attribute value, with the first attribute value being different from the second attribute value, and wherein each of the first segment and the second segment comprises a combination including both participants assigned to an experiment variant of the product and participants assigned to a control variant of the product;
• determining, based on a determination that the first segment corresponding to the first attribute value and the second segment corresponding to the second attribute value both include a segment SRM, that a root cause of the SRM is widespread within recent OCEs for the product;
• and
• before the two OCEs each compute a subsequent scorecard, stopping the two OCEs from each running the subsequent scorecard based on the root cause of the SRM being widespread within the recent OCEs for the product, wherein the subsequent scorecard is more expensive to compute than a prior scorecard.