Systems and methods for conducting and terminating a technology-assisted review

#The present invention relates generally to information processing and classification. More particularly, the present invention relates to systems, methods and computer readable media for terminating a technology-assisted review (“TAR”) process in order to efficiently classify a plurality of documents in a collection of electronically stored information.

#TAR involves the iterative retrieval and review of documents from a collection until a substantial majority (or “all”) of the relevant documents have been reviewed or at least identified. At its most general, TAR separates the documents in a collection into two classes or categories: relevant and non-relevant. Other (sub) classes and (sub) categories may be used depending on the particular application.

#Presently, TAR lies at the forefront of information retrieval (“IR”) and machine learning for text categorization. Much like with ad-hoc retrieval (e.g., a Google search), TAR's objective is to find documents to satisfy an information need, given a query. However, the information need in TAR is typically met only when substantially all of the relevant documents have been retrieved. Accordingly, TAR relies on active transductive learning for classification over a finite population, using an initially unlabeled training set consisting of the entire document population. While TAR methods typically construct a sequence of classifiers, their ultimate objective is to produce a finite list containing substantially all relevant documents, not to induce a general classifier. In other words, classifiers generated by the TAR process are a means to the desired end (i.e., an accurately classified document collection).

#The inventive principles are illustrated in the figures of the accompanying drawings which are meant to be exemplary and not limiting, and in which:

#FIG. 1 is a flow diagram illustrating a process for terminating a classification process in accordance with some embodiments of the present invention.

#FIG. 2 illustrates an exemplary gain curve for a classification process as well as other aspects for terminating a classification process in accordance with some embodiments of the present invention.

#One of the most vexing problems that has plagued the use of TAR is determining when to stop the review effort such that a sufficient number of relevant documents in the document collection have been identified. Generally, a good stopping strategy involves determining that as much relevant information as possible has been found, using reasonable effort. Certain stopping criteria for TAR processes are described in Cormack VI.

#The present invention provides a reliable method to achieve high recall using any search strategy that repeatedly retrieves documents and receives relevance feedback. A determination can be made as to when to terminate the review effort using the techniques or stopping criteria described in accordance with certain embodiments of the invention. These techniques are applicable to search strategies and classification efforts such as: ranked retrieval, interactive search and judging (“ISJ”), move-to-front pooling, and continuous active learning (“CAL”). In ISJ, a searcher repeatedly formulates queries and examines the top results from a relevance-ranking search engine. CAL, on the other hand, uses machine learning instead of, or in addition to, manually formulated queries to rank the documents for review. Techniques for carrying out these search strategies are described in Cormack VI. See e.g., Cormack VI, ¶¶65-70, 130-136, 184-190.

#One objective of the present invention is to provide quality assurance for TAR applications. Such applications include: electronic discovery (“eDiscovery”) in legal matters, systematic review in evidence-based medicine, and the creation of test collections for information retrieval (“IR”) evaluation. For these types of applications, the review effort may be measured as the total number of documents presented to a reviewer. Based on this measure, an ideal (or perfect) search would find all the relevant documents with effort equal to that number. In other words, in an ideal search, each document presented to the reviewer during the TAR process would be a relevant one. Since such an ideal search is most likely impractical or even impossible, an acceptable search strategy would find an acceptable percentage of the relevant documents and limit wasted effort (e.g., presenting non-relevant documents for review).