Techniques for deriving and/or leveraging application-centric model metric

A method of quantifying accuracy of a prediction model that has been trained on a data set parameterized by a plurality of features, the model performing in accordance with a theoretical performance manifold over an intractable input space in connection with the features, the method comprising:

• determining which of the plurality of features are strongly correlated with performance of the model;
• based on the features determined to be strongly correlated with performance of the model, creating a plurality of parameterized sub-models that, in aggregate, approximate the intractable input space;
• generating prototype exemplars for each of the created sub-models, the prototype exemplars for each created sub-model being objects to which the model can be applied to result in a match with the respective sub-model;
• and
• quantifying the accuracy of the model using the generated prototype exemplars;
• wherein the quantifying of the accuracy of the model further comprises, provided that the prototype exemplars are completely parameterized by the features:
• defining a set of one or more performance metrics for the model as a regression on the model for the prototype exemplars;
• and
• (i) approximating the expected performance of the model on a new data set as the sum of the regression on each prototype exemplar multiplied by the probability of the respective prototype exemplar matching its respective sub-model;
• and/or (ii) approximating the variance of the model on the new data set as (a) the sum of the regression on each prototype exemplar squared multiplied by the probability of the respective prototype exemplar matching its respective sub-model, subtracting (b) the square of the sum of the regression on each prototype exemplar multiplied by the probability of the respective prototype exemplar matching its respective sub-model.

A system for quantifying accuracy of a prediction model that has been trained on a data set parameterized by a plurality of features, the model performing in accordance with a theoretical performance manifold over an intractable input space in connection with the features, the system comprising:

• an electronic interface over which the model is received;
• and
• processing resources including at least one processor and a memory coupled thereto, the processing resources being configured to at least:
• determine which of the plurality of features are strongly correlated with performance of the model;
• based on the features determined to be strongly correlated with performance of the model, create a plurality of parameterized sub-models that, in aggregate, approximate the intractable input space;
• generate prototype exemplars for each of the created sub-models, the prototype exemplars for each created sub-model being objects to which the model can be applied to result in a match with the respective sub-model;
• and
• quantify the accuracy of the model using the generated prototype exemplars;
• wherein the quantifying of the accuracy of the model further comprises, provided that the prototype exemplars are representative of the intractable input space, approximating the expected performance of the model on a new data set as the sum of a set of one or more performance metrics for the model on each of the prototype exemplars multiplied by the probability of the respective prototype exemplar matching its respective sub-model.