Sample Size Determination in Sequential Hypothesis Testing
AssigneeAdobe Systems Incorporated
InventorsAlan John Malek, Yinlam Chow, Mohammad Ghavamzadeh
Life of this patent
Estimating the term…
Abstract
Sample size determination techniques in sequential hypothesis testing in a digital medium environment are described. The sample size may be determined before a test to define a number of samples (e.g., user interactions with digital marketing content) that are likely to be tested as part of the sequential hypothesis testing in order to achieve a result. The sample size may also be determined in real time to define a number of samples that likely remain for testing in order to achieve a result. The sample size may be determined in a variety of ways, such as through simulation, based on a gap between conversion rates for different options being tested, and so on.
Claims20
In a digital medium testing environment, a method implemented by at least one computing device, the method comprising:
- forming, by the at least one computing device, a plurality of samples from data describing user interactions with a plurality of items of digital content;
- determining, by the at least one computing device, conversion rates for the plurality of items of digital content from the plurality of samples;
- detecting, by the at least one computing device, a gap between the determined conversion rates for the plurality of items from the plurality of samples;
- determining, by the at least one computing device, a number of user interactions of the data to be tested as part of sequential hypothesis testing in order to reach an amount of statistical significance, the number based at least in part on the established gap;
- and
- controlling, by the at least one computing device, output of an indication that is based on the determined number of user interactions in a user interface.
In a digital medium testing environment, a method implemented by at least one computing device, the method comprising:
- generating, by the at least one computing device, a plurality of samples from data describing user interactions with first and second items of digital content;
- sequential hypothesis testing, by the at least one computing device, the plurality of samples;
- detecting, by the at least one computing device, a number of user interactions in each sample of the plurality of samples that are reached in order to achieve an amount of statistical significance for each said test as part of the sequential hypothesis testing;
- and
- determining, by the at least one computing device, a number of user interactions of the data to be tested as part of sequential hypothesis testing in order to reach the amount of statistical significance, the number based at least in part on the detected number of user interactions for each of the samples.
In a digital medium testing environment, a system comprising:
- a sampling module implemented at least partially in hardware to form a plurality of samples from data describing user interactions with first and second items of digital content;
- a sequential testing module implemented at least partially in hardware to determine conversion rates for the first and second items of digital content from the plurality of samples;
- a gap detection module implemented at least partially in hardware to detect a gap between the conversion rates for the first and second items from the plurality of samples;
- and
- an interaction determination module implemented partially in hardware to determine a number of user interactions of the data to be tested as part of sequential hypothesis testing in order to reach an amount of statistical significance, the number based at least in part on the established gap.
Dependencies are inferred from the claim text.