Orthogonal experimentation in a computing environment
AssigneeeBay Inc.
InventorsJasdeep Singh Sahni, Anil Madan, Deepak Seetharam Nadig, Po Cheung, Bhavesh Mistry, John Bodine, Michael Lo
Life of this patent
Estimating the term…
Abstract
Various embodiments include at least one of systems, methods, and software to receive input configuring tests within a computing environment to expose users to standard application or website experiences or test experiences. In some embodiments, multiple tests may be configured to run orthogonally within user experiences without affecting the results of one another. Some such embodiments preserve the ability to execute certain tests in a non-orthogonal manner while other tests are allowed to execute orthogonally.
Claims20
A system comprising:
- at least one processor;
- at least one memory device;
- a network interface device;
- and
- instructions held in the at least one memory device, the instructions executable by the at least one processor to perform a method comprising:
- receiving, over a network via the network interface device, a request for webpage related content;
- retrieving, from a database, an experiment identifier of each experiment active with regard to the received request;
- for each retrieved experiment identifier, performing a hashing function against a combination of a user identifier of a user from which the request was received and the respective experiment identifier to obtain a hash value;
- for each experiment identifier and its hash value, comparing the hash value to a traffic allotment for the experiment of the experiment identifier to determine whether the user is to be exposed to the respective experiment;
- for each experiment to which the user is to be exposed, identifying an experiment variant to which the user is to be exposed as a function of the hash value;
- for each experiment the user identifier is associated with, providing the identified experiment variant, otherwise provide a standard experience;
- and
- processing the request for the webpage related content according to any identified experiments variants.
A method comprising:
- receiving, over a network, a request for webpage related content;
- retrieving, from a database, an experiment identifier of each experiment active with regard to the received request;
- for each retrieved experiment identifier, performing a hashing function against a combination of a user identifier of a user from which the request was received and the respective experiment identifier to obtain a hash value;
- for each experiment identifier and its hash value, comparing the hash value to a traffic allotment for the experiment of the experiment identifier to determine whether the user is to be exposed to the respective experiment;
- for each experiment to which the user is to be exposed, identifying an experiment variant to which the user is to be exposed as a function of the hash value;
- for each experiment the user identifier is associated with, providing the identified experiment variant, otherwise provide a standard experience;
- and
- processing the request for the webpage related content according to any identified experiments variants.
A non-transitory computer-readable storage medium, with instructions stored thereon, which when executed by at least one processor of a computer, causes the computer to perform a method comprising:
- receiving, over a network, a request for webpage related content;
- retrieving, from a database, an experiment identifier of each experiment active with regard to the received request;
- for each retrieved experiment identifier, performing a hashing function against a combination of a user identifier of a user from which the request was received and the respective experiment identifier to obtain a hash value;
- for each experiment identifier and its hash value, comparing the hash value to a traffic allotment for the experiment of the experiment identifier to determine whether the user is to be exposed to the respective experiment;
- for each experiment to which the user is to be exposed, identifying an experiment variant to which the user is to be exposed as a function of the hash value;
- for each experiment the user identifier is associated with, providing the identified experiment variant, otherwise provide a standard experience;
- and
- processing the request for the webpage related content according to any identified experiments variants.
Dependencies are inferred from the claim text.