Long term metrics applied to multivariate testing
AssigneeNetflix, Inc.
InventorJuliette Aurisset
Life of this patent
Estimating the term…
Abstract
A set of variations of a user experience on a subscription-based online site are tested, where each of the set of variations is distinct from other variations in the set of variations. Each of a group of users is assigned to one of the set of variations. For each user request received during a predetermined window of time, one of the set of variations is determined for use in processing the user request, based on a user associated with the user request, and the respective user request is processed using the determined variation. Long-term user metric information relating to subscription services for the subscription-based online site is determined, with respect to the group of users. At least one of the set of variations is then rated based on the determined long-term user metric information.
Claims24
A method of testing a plurality of variations of a user experience on a subscription-based online site, wherein each of the plurality of variations is distinct from other variations in the plurality of variations, comprising:
- assigning each of a plurality of users to one of the plurality of variations of the user experience;
- for each of a plurality of user requests received from the plurality of users during a predetermined window of time:
- determining one of the plurality of variations for use in processing the user request, based on a user associated with the user request, and
- processing the respective user request using the determined variation;
- and
- determining long-term user metric information relating to subscription services for the subscription-based online site, with respect to the plurality of users, and for each of the plurality of variations.
A non-transitory computer-readable medium containing a program that, when executed, performs an operation for testing a plurality of variations of a user experience on a subscription-based online site, wherein each of the plurality of variations is distinct from other variations in the plurality of variations, comprising:
- assigning each of a plurality of users to one of the plurality of variations of the user experience;
- for each of a plurality of user requests received from the plurality of users during a predetermined window of time:
- determining one of the plurality of variations for use in processing the user request, based on a user associated with the user request;
- and
- processing the respective user request using the determined variation;
- and
- determining long-term user metric information relating to subscription services for the subscription-based online site, with respect to the plurality of users, and for each of the plurality of variations.
A system, comprising:
- a processor;
- and
- a memory containing a program that, when executed by the processor, performs an operation for testing a plurality of variations of a user experience on a subscription-based online site, wherein each of the plurality of variations is distinct from other variations in the plurality of variations, the operation comprising:
- assigning each of a plurality of users to one of the plurality of variations of the user experience;
- for each of a plurality of user requests received from the plurality of users during a predetermined window of time:
- determining one of the plurality of variations for use in processing the user request, based on a user associated with the user request;
- and
- processing the respective user request using the determined variation;
- and
- determining long-term user metric information relating to subscription services for the subscription-based online site, with respect to the plurality of users, and for each of the plurality of variations.
Dependencies are inferred from the claim text.