Stratified sampling applied to A/B tests

A non-transitory computer-readable medium containing a program that, when executed by one or more computer processors, performs an operation for testing a plurality of variations of a user experience, wherein each of the plurality of variations is distinct from other variations in the plurality of variations, the operation comprising:

• for each permutation of a plurality of stratification factor values for one or more stratification factors that characterize users of an online site, generating a respective plurality of pseudorandom characters, wherein each distinct character included in the plurality of pseudorandom characters represents a different variation included in the plurality of variations;
• receiving, by a computing system over a network, a plurality of user requests for accessing the online site;
• assigning the plurality of user requests to variations in the plurality of variations by computationally traversing a subset of the generated pluralities of pseudorandom characters associated with the plurality of user requests;
• tracking user retention information relating to the plurality of user requests;
• calculating a retention rate for each variation included in the plurality of variations based on the user retention information;
• and
• selecting content of the online site based on a comparison between a first retention rate of a first variation and a second retention rate of a second variation, wherein the content is associated with the first variation or the second variation.

A system, comprising:

• a processor;
• and
• a memory containing a program that, when executed by the processor, performs an operation for testing a plurality of variations of a user experience, wherein each of the plurality of variations is distinct from other variations in the plurality of variations, comprising:
• for each permutation of a plurality of stratification factor values for one or more stratification factors that characterize users of an online site, generating a respective plurality of pseudorandom characters, wherein each distinct character included in the plurality of pseudorandom characters represents a different variation included in the plurality of variations;
• receiving, by a computing system over a network, a plurality of user requests for accessing the online site;
• assigning the plurality of user requests to variations in the plurality of variations by computationally traversing a subset of the generated pluralities of pseudorandom characters associated with the plurality of user requests;
• tracking user retention information relating to the plurality of user requests;
• calculating a retention rate for each variation included in the plurality of variations based on the user retention information;
• and
• selecting content for presentation on the online site based on a comparison between a first retention rate of a first variation and a second retention rate of a second variation, wherein the content is associated with the first variation or the second variation.

A method of testing a plurality of variations of a user experience, wherein each of the plurality of variations is distinct from other variations in the plurality of variations, comprising:

• for each of a plurality of user requests, received by a computing system over a network, for accessing an online site:
• selecting a first set of pseudorandom characters in a plurality of sets of pseudorandom characters, wherein a different set of pseudorandom characters in the plurality of sets of pseudorandom characters has been generated for each permutation of the plurality of stratification factor values, and each distinct character included in the first set of pseudorandom characters represents a different variation included in the plurality of variations;
• assigning the respective user request to a variation in the plurality of variations by computationally traversing the first set of pseudorandom characters associated with the respective user request;
• monitoring users interacting with assigned variations of the plurality of variations to collect user metric information;
• evaluating, by operation of one or more computer processors, each of the variations in the plurality of variations, with respect to each of the plurality of stratification factor values;
• and
• calculating a retention rate for each of the variations in the plurality of variations;
• and
• selecting content for presentation on the online site based on a comparison between a first retention rate of a first variation and a second retention rate of a second variation, wherein the content is associated with the first variation or the second variation.