Back to results
US 10,951,503 B1Granted
Determining the validity of data collected by experiments performed at a network accessible site
AssigneeAmazon Technologies, Inc.
InventorsRui Chen, Siwei Jia, Tae Yeon Kim, Mengyun Lv, Serguei B. Stepaniants
Life of this patent
Estimating the term…
Abstract
Technologies are disclosed for determining validity of data obtained from an A/B experiment, where the experiment evaluates the desirability of a potential change at a website. The experiment is run for a period of time and based upon an expected allocation of users into the A group (control group) and the B group (e.g., treatment group), along with an actual number of users directed into the two groups, it is determined if a minimal detectable error of allocation has been exceeded and, if it has, the data is deemed to be invalid. If not, the data is deemed to be valid.
Claims20
A computer-implemented method for determining validity of data gathered from an experiment related network accessible content changes, the method comprising:
- determining a potential change to a network accessible site;
- running an experiment on network traffic at the network accessible site for an amount of time, wherein running the experiment on the network traffic comprises (i) randomly directing a first amount of the network traffic at the network accessible site into a control group where the potential change is not made to the network accessible site and (ii) randomly directing a second amount of the network traffic at the network accessible site into a treatment group where the potential change is made to the network accessible site;
- determining a first expected proportion of the network traffic to be randomly directed into the control group;
- determining a second expected proportion of the network traffic to be randomly directed into the treatment group;
- gathering interaction data related to the network traffic accessing the network accessible site, the interaction data related to at least one characteristic of behavior of the network traffic at the network accessible site during the amount of time;
- determining a first actual amount of the network traffic directed into the control group;
- determining a second actual amount of the network traffic directed into the treatment group;
- based upon the (i) first expected proportion of the network traffic to be randomly directed into the control group, the (ii) second expected proportion of the network traffic to be randomly directed into the treatment group, the (iii) first actual amount of the network traffic directed into the control group, and the (iv) second actual amount of the network traffic directed into the treatment group, computing a misallocation margin threshold for allocation of the network traffic between the control group and the treatment group;
- identifying an absolute value of an observed proportion of the network traffic directed to the control group to be greater than or equal to the misallocation margin threshold for the allocation of the network traffic;
- and
- determining that the interaction data is invalid.
8 more elements
A system comprising:
- one or more processors;
- and
- at least one computer-readable storage medium having computer-executable instructions stored thereupon which, when executed by the one or more processors, cause the system to:
- run an experiment with users related to a potential change at a network accessible site for an amount of time, wherein running the experiment with the users comprises (i) randomly directing a first amount of the users of the network accessible site into a control group where the potential change is not made to the network accessible site and (ii) randomly directing a second amount of the users of the network accessible site into a treatment group where the potential change is made to the network accessible site;
- based upon (i) expected metrics related to an expected allocation of the users between the control group and the treatment group from the randomly directing the first amount of the users of the network accessible site into the control group and the randomly directing the second amount of the users of the network accessible site into the treatment group and (ii) actual metrics related to an actual allocation of the users between the control group and the treatment group, determine that an allowed error has been exceeded by an observed proportion of the users directed to the control group;
- and
- determine that data obtained from the experiment is invalid.
4 more elements
A computer-readable storage medium having computer-executable instructions stored thereupon which, when executed by one or more processors, cause the one or more processors to:
- run an experiment with users related to a potential change at the network accessible site for an amount of time, wherein running the experiment with the users comprises randomly directing a first amount of the users at the network accessible site into a control group where the potential change is not made to the network accessible site and (ii) randomly directing a second amount of the users of the network accessible site into a treatment group where the potential change is made to the network accessible site;
- based upon (i) expected metrics related to an expected allocation of the users between the control group and the treatment group from the randomly directing the first amount of the users of the network accessible site into the control group and randomly directing the second amount of the users of the network accessible site into the treatment group and (ii) actual metrics related to an actual allocation of the users between the control group and the treatment group, determine that an allowed error has been exceeded by an observed proportion of the users directed to the control group;
- and
- determine that data obtained from the experiment is invalid.
1 more element
Dependencies are inferred from the claim text.