Systems and methods for classifying electronic information using advanced active learning techniques
OwnerGordon Villy Cormack, Maura Robin Grossmaninventors
Life of this patent
Estimating the term…
Abstract
Systems and methods for classifying electronic information or documents into a number of classes and subclasses are provided through an active learning algorithm. In certain embodiments, seed sets may be eliminated by merging relevance feedback and machine learning phases. Such document classification systems are easily scalable for large document collections, require less manpower and can be employed on a single computer, thus requiring fewer resources. Furthermore, the classification systems and methods described can be used for any pattern recognition or classification effort in a wide variety of fields, including electronic discovery in legal proceedings.
Claims18
An active learning system for processing documents in connection with conducting e-discovery in a legal proceeding, the system comprising:
- a processor being adapted to:
- generate or receive a document information profile for one or more documents in a document collection, each document information profile corresponding to a particular document and representing features of that document;
- select a first document from the document collection to present to a human reviewer;
- receive a user coding decision associated with the first document;
- calculate a classifier based on the received user coding decision associated with the first document and the document information profile for the first document;
- compute a first set of scores for a subset of documents in the collection by applying the classifier to the document information profile for each document in the subset for which the first set of scores are computed;
- determine a first plurality of rankings for documents in the document collection by choosing between the first set of scores, relevance rankings derived from user input for the documents in the document collection, or a combination of the first set of scores and the relevance rankings;
- select a further document from the document collection to present to a human reviewer based on the first plurality of rankings;
- receive a user coding decision associated with the further document;
- incrementally update the classifier based on the received user coding decision associated with the further document and the document information profile for the further document;
- compute a second set of scores for a subset of documents in the collection by applying the updated classifier to the document information profile for each document in the subset for which the second set of scores are computed;
- determine a second plurality of rankings for documents in the document collection by choosing between the second set of scores, relevance rankings derived from user input for the documents in the document collection, or a combination of the second set of scores and the relevance rankings;
- calculate an estimate of the number of documents in the document collection that are relevant to the legal proceeding by fitting scores computed for documents for which user coding decisions were received to a standard distribution;
- estimate recall based on the calculated estimate of the number of documents in the document collection that are relevant to the legal proceeding;
- in response to determining that the estimated recall is acceptable, classify a set of documents in the document collection as relevant or non-relevant to the legal proceeding using computed scores or user coding decisions;
- and
- in response to determining that the estimated recall is not acceptable, repeat the steps of selecting a further document, receiving a user coding decision associated with the further document, updating the classifier, computing a second set of scores, determining a second plurality of rankings, calculating an estimate of the number of documents in the document collection that are relevant to the legal proceeding, and estimating recall.
Dependencies are inferred from the claim text.