Document Coding Computer System and Method With Integrated Quality Assurance

A quality-assurance oriented computer-implemented method and system for automatically classifying text using machine learning comprising the following steps:

• a) eliciting user goals, classification tasks, user error tolerances and desired performance targets, analysis timelines, availability of data, feasibility of data collection, availability of computing resources, acceptable costs, and user cost constraints (step 1);
• b) presenting to users and obtaining their preferences and approvals related to a data and computing environment design including data management plan, data security plan, and high performance computing plan (step 2);
• c) collecting data needed for model training and verification, accessing individual computers or centralized servers that contain the document data, sampling documents for training and testing, labeling sampled data or re-using existing labels (step 3);
• d) performing quality control of the document data by ensuring accurate digital representation and conversions of documents to formats that are machine processable, and ensuring accurate and consistent labeling by the users through resolution of disagreements in labeling and statistical application of inter-rater reliability metrics (step 4);
• e) performing feasibility modeling, including preliminary modeling, model quality control, and model explanation (step 5);
• f) assessing class imbalance and using the information for adjusting data collection , and conducting model training and verification (step 6);
• g) optimizing the feasibility model of step 5 via additional sampling, improving labeling and classification guidelines, adding in the training corpus labeled documents from other document collections, adding unsampled labeled documents, and optionally performing model calibration and thresholding (step 7);
• h) iterating steps 1 to 6 until the user is satisfied with the model performance;
• i) producing intellectual property related documentation (step 8);
• j) verifying the optimized model of step 7 (step 9) and ensuring quality control of the verification process (step 10) by producing a second labeled sample from the data of step 3, using a predefined set of documents not used to train the model, conducting user verification of a random selection of documents from a training cross-validation step, or using a random selection of documents from the second subset randomly labeled;
• k) iterating steps 7 to 10 until the user is satisfied with the model performance;
• l) explaining the optimized and verified model by translating it to more human-accessible forms such as decision trees and rule sets and showing how the classification is made by the various classifiers;
• analyzing the predictivity of individual words and word patterns used by the model (step 11);
• m) applying the final model to the application set of documents (step 12);
• n) deploying the final model in a computational environment for continuous future use (step 13);
• o) performing additional predictive modeling and coding for additional classification tasks (step 14);
• and
• p) completing the analysis and outputting a detailed report (step 15).