Automatic ramp-up of controlled experiments
AssigneeLinkedIn Corporation
InventorsYa Xu, Weitao Duan, Shaochen Huang, Mingyue Tan, Shaohua Xie
Life of this patent
Estimating the term…
Abstract
The disclosed embodiments provide a system for managing an A/B test. During operation, the system calculates a first risk associated with ramping up exposure to a first A/B test by a first ramp amount. Next, the system uses a first sequential hypothesis test to compare the first risk with a first risk tolerance for the first A/B test. When the first sequential hypothesis test indicates that the first risk is within the first risk tolerance, the system automatically triggers a ramp-up of exposure to the first A/B test by the first ramp amount.
Claims20
A method, comprising:
- calculating, by one or more computer systems, a first risk associated with ramping up exposure to a first A/B test by a first ramp amount;
- using a first sequential hypothesis test to compare the first risk with a first risk tolerance for the first A/B test;
- and
- when the first sequential hypothesis test indicates that the first risk is within the first risk tolerance, automatically triggering, by the one or more computer systems, a ramp-up of exposure to the first A/B test by the first ramp amount.
An apparatus, comprising:
- one or more processors;
- and
- memory storing instructions that, when executed by the one or more processors, cause the apparatus to:
- calculate a first risk associated with ramping up exposure to a first A/B test by a first ramp amount;
- use a first sequential hypothesis test to compare the first risk with a first risk tolerance for the first A/B test;
- and
- when the first sequential hypothesis test indicates that the first risk is within the first risk tolerance, automatically trigger a ramp-up of exposure to the first A/B test by the first ramp amount.
A non-transitory computer-readable storage medium storing instructions that when executed by a computer cause the computer to perform a method, the method comprising:
- calculating a first risk associated with ramping up exposure to a first A/B test by a first ramp amount;
- using a first sequential hypothesis test to compare the first risk with a first risk tolerance for the first A/B test;
- and
- when the first sequential hypothesis test indicates that the first risk is within the first risk tolerance, automatically triggering a ramp-up of exposure to the first A/B test by the first ramp amount.
Dependencies are inferred from the claim text.