Classification system with methodology for efficient verification

A non-transitory computer-readable storage medium storing one or more instructions which, when executed by one or more processors, cause the one or more processors to perform steps comprising:

• obtaining a document;
• determining, using a trained classifier, a candidate label for the document from a plurality of labels;
• selecting one or more linguistic structures from the document;
• displaying a user interface that presents data from the document, including at least a portion of the one or more linguistic structures, and the candidate label, wherein the portion of the one or more linguistic structures are displayed by the user interface, wherein the user interface includes one or more user interface controls which present a first option to accept the candidate label for the document and a second option to select a different label for the document, the one or more user interface controls further presenting an element for highlighting the one or more linguistic structures within the document;
• receiving, via the one or more user interface controls, input representing selection of the first option or the second option, and further input comprising a highlighted section of the one or more linguistic structures that was important to the selection of the first option or the second option;
• associating the document with a verified label;
• changing one or more weights assigned to the highlighted section relative to a non-highlighted section during retraining of the trained classifier.

A system comprising:

• an unlabeled document database storing one or more unlabeled documents;
• a classification computer configured to:
• obtain a document from the unlabeled document database;
• determine, using a trained classifier, a candidate label for the document from a plurality of labels;
• change one or more weights assigned to a highlighted section relative to a non-highlighted section during retraining of the trained classifier;
• a verification computer configured to:
• select one or more linguistic structures from the document;
• display a user interface that presents data from the document, including at least a portion of the one or more linguistic structures, and the candidate label, wherein the portion of the one or more linguistic structures are displayed by the user interface, wherein the user interface includes one or more user interface controls which present a first option to accept the candidate label for the document and a second option to select a different label for the document, the one or more user interface controls further presenting an element for highlighting the one or more linguistic structures within the document;
• receive, via the one or more user interface controls, input representing selection of the first option or the second option, and further input comprising a highlighted section of the one or more linguistic structures that was important to the selection of the first option or the second option;
• associate the document with a verified label.