Imbalance detection in online experiments
OwnerStephane Shao, Lyubov Ignatovsky +1inventors
InventorsStephane Shao, Lyubov Ignatovsky, Julie Beckley
Life of this patent
Estimating the term…
Loading patent…
OwnerStephane Shao, Lyubov Ignatovsky +1inventors
InventorsStephane Shao, Lyubov Ignatovsky, Julie Beckley
Estimating the term…
Methods, systems, and apparatus, including computer programs encoded on computer-storage media, for imbalance detection in online experiments. In some implementations, a method includes obtaining user information indicating a first set of devices assigned a first version of a service and a second set of devices assigned a second version of the service in a multivariate testing framework; generating alternative samplings of devices; generating a threshold for detecting imbalance using the alternative samplings of devices, a generated expected first set of devices, and a generated expected second set of devices; detecting an imbalance using the obtained user information indicating the first set of devices, the second set of devices, and the generated threshold; and implementing a corrective action to mitigate the detected imbalance in the multivariate testing framework.
1-20. (canceled)
A method comprising:
A system comprising one or more computers and one or more storage devices on which are stored instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising:
One or more non-transitory computer storage media encoded with instructions that, when executed by one or more computers, cause the one or more computers to perform operations comprising:
Dependencies are inferred from the claim text.