Evaluation of predictions in the absence of a known ground truth

Abstract

Disclosed is a novel system, and method to evaluate a prediction of a possibly unknown outcome out of a plurality of predictions of that outcome. The method begins with accessing a particular prediction of an outcome out of a plurality of predictions of that outcome in which the outcome may be unknown. Next, a subsample of the plurality of predictions of the outcome is accessed. The subsample can possibly include the particular prediction. A consensus prediction of the outcome based on the subsample of the plurality of predictions is determined. A proximity of the particular prediction to the consensus prediction is determined. Each prediction is ranked out of the plurality of predictions in an order of a closest in proximity to the consensus prediction to a farthest in proximity to the consensus prediction.

Dependencies are inferred from the claim text.