Automated experiment scheduling

An apparatus comprising:

• a processor circuit;
• and
• logic to:
• commence collection of a specified number of samples of user responses to multiple versions of a user interface in a test;
• employ an alternative statistical analysis to analyze the samples as the samples are collected;
• determine whether a proposed version of the multiple versions is statistically likely to elicit a statistically significant improvement in user response over a control version of the multiple versions;
• cull the proposed version from the test to shorten a duration of the test in response to a determination that the proposed version is statistically unlikely to elicit a statistically significant improvement in user response over the control version;
• and
• complete collection of the specified number of samples and employ a classical statistical analysis to analyze the samples in response to the proposed version not eliciting the statistically significant improvement.

A computer-implemented method comprising:

• transmitting interaction data conveying different versions of a user interface to each of multiple interaction devices operated by different users to perform a test;
• commencing collection of a specified number of samples of user responses to the multiple versions of the user interface;
• analyzing the samples as the samples are collected using Bayesian analysis;
• determining whether a proposed version of the multiple versions elicits a statistically significant improvement in user response over a control version of the multiple versions;
• terminating the test before the specified number of samples are collected in response to the proposed version eliciting a statistically significant improvement over the control version;
• and
• completing collection of the specified number of samples and analyzing the samples using NHST in response to the proposed version not eliciting the statistically significant improvement.

At least one machine-readable storage medium comprising instructions that when executed by a computing device, cause the computing device to:

• transmit interaction data conveying different versions of a user interface to each of multiple interaction devices operated by different users to perform a test;
• commence collection of a specified number of samples of user responses to the multiple versions of the user interface;
• analyze the samples as the samples are collected using Bayesian analysis;
• determine whether a proposed version of the multiple versions elicits a statistically significant improvement in user response over a control version of the multiple versions;
• terminate the test before the specified number of samples are collected in response to the proposed version eliciting a statistically significant improvement over the control version;
• and
• complete collection of the specified number of samples and analyze the samples using NHST in response to the proposed version not eliciting the statistically significant improvement.