Methods, systems, and media for identifying errors in predictive models using annotators

A system for evaluating predictive models in classification systems, the system comprising:

• a processor that:
• causes an input region to be presented to a user, wherein the input region receives an instance from the user;
• in response to receiving the instance from the user, retrieves a classification assigned by a predictive model for the received instance and a confidence value associated with the classification, wherein the classification has been stored in association with the instance from the user in response to determining that the classification exceeds a threshold value, and wherein the threshold value is set based at least in part on a penalty associated with misclassification of the instance;
• determines whether the received instance has been incorrectly classified by the predictive model;
• in response to determining that the received instance has been incorrectly classified by the predictive model, (i) causing, using the hardware processor, the received instance and a corrected classification provided by the user to be used as training data to update the predictive model, and (ii) determines a reward associated with the incorrect classification made by the predictive model, wherein the reward is directly proportional to the confidence value associated with the classification of the received instance;
• updating, using the hardware processor, the predictive model using the received instance as training data;
• and
• causes an indication of the reward to be provided to the user.

A non-transitory computer-readable medium containing computer-executable instructions that, when executed by a processor, cause the process to perform a method for evaluating predictive models in classification systems, the method comprising:

• causing an input region to be presented to a user, wherein the input region receives an instance from the user;
• in response to receiving the instance from the user, retrieving a classification assigned by a predictive model for the received instance and a confidence value associated with the classification, wherein the classification has been stored in association with the instance from the user in response to determining that the classification exceeds a threshold value, and wherein the threshold value is set based at least in part on a penalty associated with misclassification of the instance;
• determining whether the received instance has been incorrectly classified by the predictive model;
• in response to determining that the received instance has been incorrectly classified by the predictive model, (i) causing, using the hardware processor, the received instance and a corrected classification provided by the user to be used as training data to update the predictive model, and (ii) determining a reward associated with the incorrect classification made by the predictive model, wherein the reward is directly proportional to the confidence value associated with the classification of the received instance;
• updating, using the hardware processor, the predictive model using the received instance as training data;
• and
• causing an indication of the reward to be provided to the user.