US 9,854,029 B1Granted

Systems for determining improper assignments in statistical hypothesis testing

AssigneeAmazon Technologies, Inc.

InventorsSebastian Kohlmeier, Rusli Ciandy, Rui Chen, Sameer Rajendra Rajyaguru

Life of this patent

Estimating the term…

Abstract

Described are techniques for performing statistical hypothesis experiments and determining user responses that do not match an assigned treatment. One of multiple experiment states may be assigned to a set of client devices. Response data that includes indications of the experiment state that was executed may be compared to the assigned experiment state to determine allocation matches and allocation mismatches. The allocation mismatches may be separated based on the assigned and executed experiment states to determine mismatches caused by preexisting content stored on a client device and mismatches caused by other experiment parameters.

Claims20

Dependent claims

A system comprising:

a server in communication with a plurality of client devices, the server comprising a hardware processor, memory, and computer-executable instructions stored in the memory that, when executed by the hardware processor, cause the hardware processor to:
establish communication sessions with the plurality of client devices, wherein at least a portion of the communication sessions comprise a session identifier;
assign a control state to a first subset of the plurality of client devices and a treatment state to a second subset of the plurality of client devices, wherein at least one of the control state or the treatment state causes output of one or more features by a respective client device by executing native code on the respective client device, further wherein the native code is compiled to execute on a hardware processor of the respective client device;
store assigned experiment states for the first subset and the second subset of the plurality of client devices in association with a respective session identifier;
receive response data from one or more first client devices of the first subset and one or more second client devices of the second subset, wherein the response data comprises session identifiers associated with the one or more first client devices and the one or more second client devices and state identifiers indicative of the output of the control state or the treatment state by the one or more first client devices and the one or more second client devices;
compare at least a subset of the state identifiers to at least a subset of the assigned experiment states to determine a result comprising:
an allocation match;
an allocation mismatch associated with assignment of the treatment state and response data associated with the control state;
and
an allocation mismatch associated with assignment of the control state and response data associated with the treatment state;
generate statistical data, using the result, wherein the statistical data is indicative of allocation matches corresponding to the control state and allocation matches corresponding to the treatment state, wherein the statistical data is not indicative of the allocation mismatches associated with assignment of the treatment state and response data associated with the control state and the allocation mismatches associated with assignment of the control state and response data associated with the treatment state;
and
cause modification of an experiment parameter based at least in part on the allocation mismatches associated with assignment of the control state and response data associated with the treatment state.

10 more elements

A method comprising:

assigning a first experiment state comprising a first state identifier to a first subset of a plurality of client devices;
assigning a second experiment state comprising a second state identifier to a second subset of the plurality of client devices;
storing an indication of assigned experiment states for the first subset and the second subset;
receiving response data from the plurality of client devices, wherein the response data comprises at least one indication of the first state identifier and at least one indication of the second state identifier;
determining, from the response data and the assigned experiment states, one or more of:
at least one allocation match corresponding to assignment of the first experiment state and response of the first experiment state;
at least one allocation match corresponding to assignment of the second experiment state and response of the second experiment state;
at least one allocation mismatch corresponding to assignment of the first experiment state and response of the second experiment state;
or
at least one allocation mismatch corresponding to assignment of the second experiment state and response of the first experiment state;
generating statistical data indicative of one or more of:
the at least one allocation match corresponding to assignment of the first experiment state and response of the first experiment state;
or
the at least one allocation match corresponding to assignment of the second experiment state and response of the second experiment state;
and
causing modification of an experiment parameter based at least in part on one or more of:
the at least one allocation mismatch corresponding to assignment of the first experiment state and response of the second experiment state;
or
the at least one allocation mismatch corresponding to assignment of the second experiment state and response of the first experiment state.

16 more elements

A non-transitory computer-readable medium storing instructions that, when executed, cause a hardware processor to perform a method of:

accessing communication sessions with a plurality of client devices, wherein at least a subset of the plurality of client devices includes data associated with a first experiment state having a first state identifier, wherein the first experiment state causes the at least a subset of the plurality of client devices to generate output associated with a first set of characteristics;
assigning a second experiment state to a first subset of the plurality of client devices, wherein the second experiment state is associated with a second state identifier and causes the first subset of the plurality of client devices to generate output associated with a second set of characteristics;
assigning a third experiment state to a second subset of the plurality of client devices, wherein the third experiment state is associated with a third state identifier and causes the second subset of the plurality of client devices to generate output associated with the first set of characteristics associated with the first experiment state;
generating assignment data indicating assignments of the second experiment state to the first subset and assignments of the third experiment state to the second subset;
receiving, from the plurality of client devices, response data comprising at least one indication of the first experiment state, at least one indication of the second experiment state, and at least one indication of the third experiment state;
determining, using the response data and the assignment data, one or more of:
at least one allocation match corresponding to the second experiment state;
at least one allocation match corresponding to the third experiment state;
at least one allocation mismatch corresponding to assignment of the second experiment state and response of the third experiment state;
at least one allocation mismatch corresponding to assignment of the third experiment state and response of the second experiment state;
or
determining, using the response data, at least one allocation mismatch corresponding to response of the first experiment state;
generating statistical data indicative of one or more of:
the at least one allocation match corresponding to the second experiment state;
or
the at least one allocation match corresponding to the third experiment state;
and
causing modification of an experiment parameter based on one or more of:
the at least one allocation mismatch corresponding to assignment of the second experiment state and response of the third experiment state;
the at least one allocation mismatch corresponding to assignment of the third experiment state and response of the second experiment state;
or
the at least one allocation match corresponding to the second experiment state.

19 more elements

Dependencies are inferred from the claim text.

patentgrep

Loading patent…

Abstract

A system comprising:

a server in communication with a plurality of client devices, the server comprising a hardware processor, memory, and computer-executable instructions stored in the memory that, when executed by the hardware processor, cause the hardware processor to:
establish communication sessions with the plurality of client devices, wherein at least a portion of the communication sessions comprise a session identifier;
assign a control state to a first subset of the plurality of client devices and a treatment state to a second subset of the plurality of client devices, wherein at least one of the control state or the treatment state causes output of one or more features by a respective client device by executing native code on the respective client device, further wherein the native code is compiled to execute on a hardware processor of the respective client device;
store assigned experiment states for the first subset and the second subset of the plurality of client devices in association with a respective session identifier;
receive response data from one or more first client devices of the first subset and one or more second client devices of the second subset, wherein the response data comprises session identifiers associated with the one or more first client devices and the one or more second client devices and state identifiers indicative of the output of the control state or the treatment state by the one or more first client devices and the one or more second client devices;
compare at least a subset of the state identifiers to at least a subset of the assigned experiment states to determine a result comprising:
an allocation match;
an allocation mismatch associated with assignment of the treatment state and response data associated with the control state;
and
an allocation mismatch associated with assignment of the control state and response data associated with the treatment state;
generate statistical data, using the result, wherein the statistical data is indicative of allocation matches corresponding to the control state and allocation matches corresponding to the treatment state, wherein the statistical data is not indicative of the allocation mismatches associated with assignment of the treatment state and response data associated with the control state and the allocation mismatches associated with assignment of the control state and response data associated with the treatment state;
and
cause modification of an experiment parameter based at least in part on the allocation mismatches associated with assignment of the control state and response data associated with the treatment state.

10 more elements