Computer System and Method That Determines Sample Size and Power Required For Complex Predictive and Causal Data Analysis

A computer-implemented method and system for determining sample size and power required for complex predictive and causal data analysis comprising the following steps:

• a) accepting as inputs a dataset D, the known or estimated prior distribution P of the target response variable in D, a desired performance A0 and a desired certainty level L that the desired performance will be attained by analyzing D with the sample size recommended by the system;
• b) a preparatory phase of knowledge base creation consisting of:
• 1) compiling a collection of datasets that have a variety of prior distributions for the target class and are otherwise broadly representative of the types of data relevant to the modeling to be accomplished;
• 2) for every dataset in the dataset collection, generating random samples of increasing sample sizes;
• 3) for each random sample training a model using the learning method(s) of choice and estimating performance;
• 4) saving performance values of each of the samples for all sample sizes;
• c) a first part analysis phase consisting of:
• 1) selecting a subset of the datasets from the knowledge base that have prior approximately equal to prior distribution P;
• 2) examining the distribution of performances for the datasets selected in step c.1 to determine the minimal sample size S1 such that at least L % datasets have desired performance A0 or better;
• 3) obtaining a random sample TRAIN of size S1 from D;
• 4) training a model and estimating performance A1 by cross-validation of other performance estimators;
• 5) if A1>=A0, training the classifier in all labeled data in TRAIN and outputting the model, and terminating;
• d) a second-part analysis phase that is activated if A1 in the first-part analysis phase is less than A0, and comprising:
• 1) selecting the subset of the datasets from the knowledge base that have prior equal to the prior distribution P and having performance A1 at sample size S1;
• 2) finding the smallest sample size S2>S1 that achieves performance=A0 in at least L of the datasets identified in step d.1;
• 3) obtaining a random sample TRAIN of data from D of sample size S2;
• 4) training a model and estimating performance A2 by using cross-validation of other performance estimators;
• 5) if A2>=A0, training the classifier in all labeled data from TRAIN, outputting the model, and terminating;
• and
• 6) if A2<A0 then reiterating second-part analysis from step d.1 using the new performance estimate A2 instead of A1 until A0 is reached or until a maximum number of iterations is carried out or until there are no datasets left in step d.1.