A computer implemented method for determining an optimal sample size for experiments targeting units having specific static criteria, the method comprising:
- maintaining access, by a computer, to a plurality of units of a production set, wherein the units comprise mobile computing devices that are associated with empirically measurable activity, such that the empirically measurable activity can be influenced by using specific treatments mapped to a specific goal, wherein the production set comprises a plurality of units that are being exposed to a plurality of treatments over time and corresponding empirical results are being measured and tracked, wherein the treatments comprise electronic messages automatically transmitted by a sample size and treatment optimization system to the mobile computing devices;
- selecting, by the computer, at a specific point in time, a subset of units of the production set meeting specific static criteria, the selected subset comprising a given number of units, the given number having been determined for creating a treatment group and a control group;
- creating, by the computer, a treatment group of units from the subset meeting the specific static criteria and a control group of units from the subset meeting the specific static criteria;
- performing, by the computer, a paired comparison test on the treatment group and the control group, wherein the paired comparison test compares empirical results for the units of the treatment group to empirical results for the units of the control group, the empirical results for the units of the treatment group and for the units of control group being against the specific goal over time, measured at the specific point in time at which the units were selected from the production set;
- accepting, by the computer, the treatment group and the control group only in response to a mean percentage difference between the treatment group and the control group being less than a specific threshold value;
- calculating, by the computer, expected signal-to-noise ratio of the units of the treatment group and the units of control group according to a specific signal-to-noise ratio calculation rule set, taking into account empirical results for the units of the treatment group and for the units of control group against the specific goal over time, measured at the specific point in time at which the units were selected from the production set;
- performing, by the computer, a test of the calculated expected signal-to-noise ratio of the units of the treatment group and the units of control group;
- accepting, by the computer, the treatment group and the control group only in response to the calculated expected signal-to-noise ratio exceeding a specific threshold value;
- recording, by the computer, mean of the control group, mean of the treatment group, pairwise standard deviation of the treatment and control groups, and pairwise mean of the treatment and control groups, calculated based on measured empirical results for the units of the treatment group and for the units of the control group against the specific goal over time, measured at the specific point in time at which the units were selected from the production set;
- performing, by the computer, an experiment comprising exposing the units of the treatment group to a specific treatment and not exposing the units of the control group to the specific treatment;
- calculating, by the computer, an effect of the performed experiment as mean difference in measured activity between the units of the treatment group and the units of the control group after having performed the experiment;
- recording, by the computer, the calculated effect of the performed experiment;
- and
- calculating, by the computer, an optimal sample size to use for subsequent experiments targeting units having the specific static criteria, according to a specific sample size calculation rule set taking into account target effect of the performed experiment and an updated expected signal-to-noise ratio, wherein the updated expected signal-to-noise ratio is calculated based on measured empirical results for the units of the treatment group and for the units of the control group against the specific goal over time, measured at a second specific point in time occurring after the performing of the experiment.