Detecting false positives in statistical models

A system for estimating whether a statistical model selected from a plurality of statistical models trained using observed historical data is a false positive, comprising:

• at least one processor executing a code, the code comprising:
• code instructions to receive a plurality of predicted outcomes computed by a plurality of statistical models for a historical dataset comprising a plurality of past observations;
• code instructions to compute a correlation matrix for the plurality of predicted outcomes;
• code instructions to cluster the plurality of predicted outcomes in a plurality of clusters according to a plurality of clustering schemes based on the correlation matrix, each of the plurality of clustering schemes defines a different number of clusters;
• code instructions to select a clustering scheme which achieves a highest quality score among a plurality of quality scores computed for the plurality of clustering schemes;
• code instructions to compute, for each of the clusters of the selected clustering scheme, an aggregated predicted outcome which aggregates the predicted outcomes clustered in the respective cluster;
• code instructions to compute an estimated variance across the clusters of the selected clustering scheme;
• and
• code instructions to compute a false positive probability of a selected one of the plurality of statistical models based on the aggregated predicted outcome of the cluster comprising the predicted outcome computed by the selected statistical model, the number of clusters in the selected clustering scheme, and the estimated variance across all clusters in the selected clustering scheme.

A method of estimating whether a selected statistical model trained using observed historical data is a false positive, comprising:

• using at least one processor for:
• receiving a historical dataset comprising a plurality of past observations ordered along a past time flow;
• partitioning the plurality of observations to a plurality of groups each comprising a respective subset of the plurality of past observations;
• creating a plurality of combinatorial train-test sets each comprising the plurality of groups in a unique training-testing split in which at least some of the plurality of groups are included in a respective testing set and a reminder of the plurality of groups are included in a respective training set, the observations in the groups of each testing set are at least partially purged with respect to the observations in the groups of the respective training set;
• receiving a plurality of predicted outcomes, each computed by applying an evaluated statistical model trained with the training set of a respective one the plurality of combinatorial train-test sets to the testing set of the respective combinatorial train-test set;
• creating a plurality of virtual past time flows by aggregating the plurality of predicted outcomes;
• and
• estimating whether the evaluated statistical model applied with at least one rule is a false based on a distribution of performance scores computed on the plurality of virtual past time flows.

A system estimating whether a selected statistical model trained using observed historical data is a false positive, comprising:

• at least one processor executing a code, the code comprising:
• code instructions to receive historical dataset comprising a plurality of past observations ordered along a past time flow;
• code instructions to partition the plurality of observations to a plurality of groups each comprising a respective subset of the plurality of past observations;
• code instructions to create a plurality of combinatorial train-test sets each comprising the plurality of groups in a unique training-testing split in which at least some of the plurality of groups are included in a respective testing set and a reminder of the plurality of groups are included in a respective training set, the observations in the groups of each testing set are at least partially purged with respect to the observations in the groups of the respective training set;
• code instructions to receive a plurality of predicted outcomes each computed by applying an evaluated statistical model trained with the training set of a respective one the plurality of combinatorial train-test sets to the testing set of the respective combinatorial train-test set;
• code instructions to construct a plurality of virtual past time flows by aggregating the plurality of predicted outcomes;
• and
• code instructions to estimate whether the evaluated statistical model applied with at least one rule is a false based on a distribution of performance scores computed on the plurality of virtual past time flows.