Stratified sampling applied to a/b tests
AssigneeNetflix, Inc.
InventorBryan Jacob Gumm
Life of this patent
Estimating the term…
Abstract
Embodiments provide techniques for testing a plurality of variations of a user experience, where each of the plurality of variations is distinct from other variations in the plurality of variations. For each permutation of a plurality of stratification factors that characterize users of the online site, embodiments generate a respective set of pseudorandom numbers. Additionally, user requests are assigned to variations in the plurality of variations, based on the generated sets of pseudorandom numbers. Embodiments also track user retention information relating to the user requests. At least one of the plurality of variations is rated, based on the user retention information.
Claims22
A method of testing a plurality of variations of a user experience, wherein each of the plurality of variations is distinct from other variations in the plurality of variations, comprising:
- for each permutation of a plurality of stratification factor values for one or more stratification factors that characterize users of the online site, generating a respective set of pseudorandom characters;
- assigning user requests to variations in the plurality of variations, based on the generated sets of pseudorandom characters;
- tracking, by operation of one or more computer processors, user retention information relating to the user requests;
- and
- rating at least one of the plurality of variations, based on the user retention information.
A non-transitory computer-readable medium containing a program that, when executed by one or more computer processors, performs an operation for testing a plurality of variations of a user experience, wherein each of the plurality of variations is distinct from other variations in the plurality of variations, the operation comprising:
- for each permutation of a plurality of stratification factor values for one or more stratification factors that characterize users of the online site, generating a respective set of pseudorandom characters;
- assigning user requests to variations in the plurality of variations, based on the generated sets of pseudorandom characters;
- tracking user retention information relating to the user requests;
- and
- rating at least one of the plurality of variations, based on the user retention information.
A system, comprising:
- a processor;
- and
- a memory containing a program that, when executed by the processor, performs an operation for testing a plurality of variations of a user experience, wherein each of the plurality of variations is distinct from other variations in the plurality of variations, comprising:
- for each permutation of a plurality of stratification factor values for one or more stratification factors that characterize users of the online site, generating a respective set of pseudorandom characters;
- assigning user requests to variations in the plurality of variations, based on the generated sets of pseudorandom characters;
- tracking user retention information relating to the user requests;
- and
- rating at least one of the plurality of variations, based on the user retention information.
A method of testing a plurality of variations of a user experience, wherein each of the plurality of variations is distinct from other variations in the plurality of variations, comprising:
- for each of a plurality of user requests:
- determining a stratification factor value, from a plurality of stratification factor values, that is associated with the respective user request;
- and
- assigning the respective user request to one of the plurality of variations, based on the determined stratification factor value;
- monitoring users interacting with assigned variations of the plurality of variations to collect user metric information;
- and
- evaluating, by operation of one or more computer processors, each of the variations in the plurality of variations, with respect to each of the plurality of stratification factor values.
Dependencies are inferred from the claim text.