Imbalance detection in online experiments

A non-transitory computer-readable medium storing one or more instructions executable by a computer system to perform operations comprising:

• obtaining user information indicating a first set of devices assigned a first version of a service and a second set of devices assigned a second version of the service in a multivariate testing framework;
• generating alternative samplings of devices assigned to the first version and the second version of the service;
• generating an expected first set of devices assigned to the first version of the service and an expected second set of devices assigned to the second version of the service;
• generating a threshold for detecting imbalance using the alternative samplings of devices, the expected first set of devices, and the expected second set of devices;
• detecting an imbalance using the obtained user information indicating the first set of devices, the second set of devices, and the generated threshold;
• and
• implementing a corrective action to mitigate the detected imbalance in the multivariate testing framework.

A system, comprising:

• one or more processors;
• and
• machine-readable media interoperably coupled with the one or more processors and storing one or more instructions that, when executed by the one or more processors, perform operations comprising:
• obtaining user information indicating a first set of devices assigned a first version of a service and a second set of devices assigned a second version of the service in a multivariate testing framework;
• generating alternative samplings of devices assigned to the first version and the second version of the service;
• generating an expected first set of devices assigned to the first version of the service and an expected second set of devices assigned to the second version of the service;
• generating a threshold for detecting imbalance using the alternative samplings of devices, the expected first set of devices, and the expected second set of devices;
• detecting an imbalance using the obtained user information indicating the first set of devices, the second set of devices, and the generated threshold;
• and
• implementing a corrective action to mitigate the detected imbalance in the multivariate testing framework.