Online product testing using bucket tests

A method, comprising:

• storing, in threshold metric circuitry, a threshold metric of a bucket test of an update to an online product, wherein the threshold metric includes a software metric associated with the online product;
• storing, in minimum difference circuitry, a predetermined minimum difference of the threshold metric;
• storing, in control circuitry, a control metric;
• running, by test-running circuitry, a one-sided bucket test using the threshold metric, the predetermine minimum difference, and the control metric, which results in a test conclusion;
• storing, by confidence circuitry, the test conclusion;
• and
• providing, by launch circuitry, the update to the online product.

A method, comprising:

• selecting, by bucket testing circuitry, a primary attribute according to analytics;
• determining, by the circuitry, whether to consider a secondary attribute according to the analytics;
• selecting, by the circuitry, a one-sided test with a minimum difference for the primary attribute according to the determination of whether to consider the secondary attribute;
• and
• running, by the circuitry, the one-sided test with the minimum difference using the primary attribute as a threshold metric.