Sample delta monitoring
AssigneeeBay Inc.
InventorsKeyu Nie, Ted Tao Yuan, Bingquan Xu
Life of this patent
Estimating the term…
Abstract
Sample delta monitoring is described herein. In one or more implementations, first and second experimental groups are defined in a randomized controlled trial. A first set of data indicative of a first experimental group in the randomized controlled trial is received. A second set of data indicative of a second experimental group in the randomized controlled trial is received. Based on sample counts of the first and second sets of data, an indication is generated that the experimental groups in the randomized controlled trial are skewed.
Claims20
A method of monitoring variations in a randomized controlled trial, the method comprising:
- defining first and second experimental groups of online users in the randomized controlled trial;
- performing the randomized controlled trial, at least in part, by delivering different versions of digital content to the first and second experimental groups of online users;
- receiving a first set of data indicative of responses to a first version of digital content by a first experimental group in the randomized controlled trial;
- receiving a second set of data indicative of responses to a second version of digital content by a second experimental group in the randomized controlled trial;
- calculating a delta value between the first and second sets of data using a sample delta monitoring algorithm;
- based on the delta value between the first and second sets of data, generating an indication that at least one of the experimental groups in the randomized controlled trial are skewed;
- and
- based on the indication, halting the randomized controlled trial.
A computing system, comprising:
- one or more processors;
- and
- a computer-readable storage medium having computer-executable instructions stored thereupon which, when executed by the processor, cause the processor to perform operations including:
- performing a randomized controlled trial, at least in part, by delivering different versions of digital content to first and second experimental groups of online users;
- receiving a first set of data indicative of responses to a first version of digital content by a first experimental group in a randomized controlled trial;
- receiving a second set of data indicative of responses to a second version of digital content by a second experimental group in the randomized controlled trial;
- calculating a delta value between the first and second sets of data using a sample delta monitoring algorithm;
- based on the delta value between the first and second sets of data, generating an indication that at least one of the experimental groups in the randomized controlled trial are skewed;
- and
- based on the indication, halting the randomized controlled trial.
A computer-readable storage medium having computer-executable instructions stored thereupon which, when executed by a processor of a computing device, cause the computing device to:
- define first and second experimental groups of online users in a randomized controlled trial;
- perform the randomized controlled trial, at least in part, by delivering different versions of digital content to the first and second experimental groups of online users;
- receive a first set of data indicative of responses to a first version of digital content by a first experimental group in the randomized controlled trial;
- receive a second set of data indicative of responses to a second version of digital content by a second experimental group in the randomized controlled trial;
- calculate a delta value between the first and second sets of data using a sample delta monitoring algorithm;
- based on the delta value between the first and second sets of data, generate an indication that at least one of the experimental groups in the randomized controlled trial are skewed;
- and
- based on the indication, halt the randomized controlled trial.
Dependencies are inferred from the claim text.