System for enhancing expert-based computerized analysis of a set of digital documents and methods useful in conjunction therewith

An electronic document analysis system operative for receiving N electronic documents pertaining to a case encompassing a set of issues including at least one issue and establishing relevance of at least the N electronic documents to at least one individual issue in the set of issues, the system comprising:

• a processor operative, for at least one individual issue from among said set of issues, for:
• i. receiving an output of a categorization process applied to documents in at least control subsets of said at least N electronic documents, said output including, for each document in said subsets, one of a relevant-to-said-individual issue indication and a non-relevant-to-said-individual issue indication;
• ii. seeking an input as to whether or not to initiate a new iteration I;
• if a new iteration is initiated, perform steps iii-x;
• and if a new iteration is not initiated, go to step xi;
• iii. selecting m electronic documents from among a subset of the N electronic documents that are not in the control set and that were not used in previous rounds for training the classifier;
• iv. receiving an output of a categorization process applied to the m electronic documents;
• v. adding the m electronic documents to an existing training subset and building a text classifier simulating said categorization process using said output for all documents in said training subset of documents;
• vi. evaluating said text classifier's quality using said output for documents in said control subset;
• vii. selecting a cut-off point for binarizing said rankings of said documents in said control subset;
• viii. using said cut-off point, computing and storing at least one quality criterion characterizing said binarizing of said rankings of said documents in said control subset, thereby to define a quality of performance indication of a current iteration I;
• ix. displaying a comparison of the quality of performance indication of the current iteration I to quality of performance indications of previous iterations;
• x. returning to step ii;
• and
• xi. generating a computer display of said output of said categorization process received in step i as most recently performed, including, for each document in said subsets, one of a relevant-to-said-individual issue indication and a non-relevant-to-said-individual issue indication.

A computer program product, comprising a non-transitory computer readable medium having computer readable program code embodied therein, said computer readable program code adapted to be executed to implement a method for electronic document analysis operative for receiving N electronic documents pertaining to a case encompassing a set of issues including at least one issue and establishing relevance of at least the N electronic documents to at least one individual issue in the set of issues, said method comprising:

• i. receiving an output of a categorization process applied to documents in at least control subsets of said at least N electronic documents, said output including, for each document in said subsets, one of a relevant-to-said-individual issue indication and a non-relevant-to-said-individual issue indication;
• ii. seeking an input as to whether or not to initiate a new iteration I. If so perform steps iii-x;
• if not go to step xi;
• iii. selecting m documents from among a subset of the N documents that are not in the control set and that were not used in previous rounds for training the classifier;
• iv. receiving an output of a categorization process applied to the m documents;
• v. adding the m documents to an existing training subset and building a text classifier simulating said categorization process using said output for all documents in said training subset of documents;
• vi. evaluating said text classifier's quality using said output for documents in said control subset;
• vii. selecting a cut-off point for binarizing said rankings of said documents in said control subset;
• viii. using said cut-off point, computing and storing at least one quality criterion characterizing said binarizing of said rankings of said documents in said control subset, thereby to define a quality of performance indication of a current iteration I;
• ix. displaying a comparison of the quality of performance indication of the current iteration I to quality of performance indications of previous iterations;
• x. returning to step ii;
• and
• xi. generating a computer display of said output of said categorization process received in step i as most recently performed, including, for each document in said subsets, one of a relevant-to-said-individual issue indication and a non-relevant-to-said-individual issue indication.