System and method for pre-indexing filtering and correction of documents in search systems
AssigneeOpen Text Holdings, Inc.
InventorsBruce Edward Kiefer, Gregory John Berka
Life of this patent
Estimating the term…
Abstract
Embodiments as disclosed herein provide a search system with an pre-indexing filter that provides both a sophisticated and contextually tailored approach to filtering documents and a corrector that is adapted to alter a document that has been designated to be filtered out from the indexing process and determine if the altered document should be indexed. The alteration of the document may be tied to the attributes, rules or thresholds used to initially filter the document from the indexing process. The filtering criteria can thus be tailored to a specific context such that both the initial filtering and the alteration process may be better suited for application in that context.
Claims21
A search system, comprising:
- a processor;
- a data store, having an index of a corpus stored thereon, wherein the corpus comprises a set of documents;
- and
- a non-transitory computer readable medium, having instructions executable on the processor for:
- receiving a first set of tokens for a document from a text extractor;
- providing the set of tokens to a detector for determining if each of the received first set of tokens is of an associated type;
- determining an associated detector score based on the determination if each of the received first set of tokens is of the associated type of token, the detector score associated with the associated type;
- determining a filter score based on the detector score produced by the detector, wherein the filter score is based on a scoring rule associated with the detector or the associated type;
- determining whether the document should be indexed based on the application of a verdict rule, wherein the verdict rule includes an expression for evaluating the filter score associated with the detector;
- and
- when it is determined that the document should be indexed, providing the document to an indexer adapted to index the first set of tokens for the document.
A non-transitory computer readable medium, comprising instructions for:
- receiving a first set of tokens for a document from a text extractor, wherein the document is one of a corpus of documents associated with an index;
- providing the set of tokens to a detector for determining if each of the received first set of tokens is of an associated type;
- determining an associated detector score based on the determination if each of the received first set of tokens is of the associated type of token, the detector score associated with the associated type;
- determining a filter score based on the detector score produced by the detector, wherein the filter score is based on a scoring rule associated with the detector or the associated type;
- determining whether the document should be indexed based on the application of a verdict rule, wherein the verdict rule includes an expression for evaluating the filter score associated with the detector;
- and
- when it is determined that the document should be indexed, providing the document to an indexer adapted to index the first set of tokens for the document.
A method, comprising:
- receiving a first set of tokens for a document from a text extractor, wherein the document is one of a corpus of documents associated with an index;
- providing the set of tokens to a detector for determining if each of the received first set of tokens is of an associated type;
- determining an associated detector score based on the determination if each of the received first set of tokens is of the associated type of token, the detector score associated with the associated type;
- determining a filter score based on the detector score produced by the detector, wherein the filter score is based on a scoring rule associated with the detector or the associated type;
- determining whether the document should be indexed based on the application of a verdict rule, wherein the verdict rule includes an expression for evaluating the filter score associated with the detector;
- and
- when it is determined that the document should be indexed, providing the document to an indexer adapted to index the first set of tokens for the document.
Dependencies are inferred from the claim text.