Determining near duplicate "noisy" data objects
AssigneeEquivio Ltd.
InventorsYiftach Ravid, Amir Milo
Life of this patent
Estimating the term…
Abstract
A system configured to find near duplicate documents. For each two (or more) documents that are similar to each other, the system is configured to identify which of the differences is likely to be generated by an Optical Character Recognition software or otherwise due to difference between the original documents. As a result, the process of identifying similarity between documents is improved by identifying documents that were originally exact duplicates but are different one with respect to the other only due to OCR errors, or correct the similarity level between the documents by correcting errors introduced by the OCR tool.
Claims54
A method for constructing a glossary operable to mask noise features in at least two documents that are similar in accordance with a criterion;
- the noise features being nominally identical features that appear differently in the documents, the method comprising:
- a) providing a discrepancy list for the at least two documents that includes the noise features;
- and
- b) constructing a glossary for masking the noise features utilizing at least the discrepancy list.
A method for masking noise errors in at least two documents wherein nominally identical features appear differently, the method comprising:
- a) maintaining a glossary for mapping noise features in any of the documents to common mapped features;
- b) comparing respective pairs of features in at least two documents;
- for each pair of features that differ by less than a predetermined threshold, if none of the features appear in the glossary, mapping said pair of features to a common mapped feature and automatically adding the mapped feature to the glossary;
- or
- if at least one of the pair of features appears in the glossary, extracting from the glossary the common mapped feature and automatically replacing said feature by the common mapped feature.
A method for masking noise features in a document wherein, owing to an error, at least one feature in the document is misrepresented and appears as a noise feature, the method comprising:
- a) maintaining a glossary that maps noise features in documents to common mapped features;
- b) identifying at least one candidate document that is similar by at least a given threshold to said document, and for every candidate document, determining a discrepancy list that includes noise features being nominally identical features that appear differently in the document and the candidate document;
- c) utilizing the discrepancy list for comparing corresponding features that originate from the document and the candidate document respectively, and in the case that the corresponding features differ in less than a predetermined threshold, if the corresponding features appear in the glossary, automatically masking said features by replacing said corresponding features by said common mapped feature.
A method for determining whether at least one document A is a candidate for near duplicate to a document B;
- at least two of said documents include noise features being nominally identical features that appear differently in the documents;
- the method comprising:
- (a) maintaining (i) a glossary configured to map noise features to common mapped features and (ii) a repository storing documents A;
- (b) applying a near-duplicate technique to document B and to at least one of said stored documents A, for determining at least one document A that is candidate for near duplicate to document B.
- (c) for each one of at least one of said candidate documents A,
- (i) utilizing the glossary, for said candidate document A and document B, for masking noise features, if any, by mapping said noise features into common mapped features, giving rise to masked documents;
- (ii) determine resemblance between said masked documents.
A method for determining whether at least one masked document A is a candidate for near duplicate to a document B;
- at least two of said documents include noise features being nominally identical features that appear differently in the documents;
- the method comprising:
- (a) maintaining (i) a glossary configured to map noise features to common mapped features, and (ii) a repository storing masked documents A;
- (b) utilizing the glossary, for said document B, for masking noise features, if any, by mapping said noise features into common mapped features, giving rise to masked document B;
- (c) applying a near-duplicate technique to the masked documents, for determining at least one masked document A that is candidate for near duplicate to the masked document B;
- (d) for each one of at least one of said candidate documents A,
- (i) utilizing the glossary, for said candidate document A and masked document B, for masking noise features, if any, by mapping said noise features into common mapped features, giving rise to masked documents;
- (ii) determine resemblance between said masked documents;
- and above threshold,
- (iii) identifying, in said document and masked document B, at least one pair of noise features, if any, which do not appear in the glossary, and in respect of each pair, mapping said pair of features to a respective common mapped feature and automatically adding said mapped feature to the glossary.
A system for constructing a glossary operable to mask noise features in at least two documents that are similar in accordance with a criterion;
- the noise features being nominally identical features that appear differently in the documents, the system comprising:
- means for providing a discrepancy list (84,85) for the at least two documents that include the noise features;
- and
- constructing system for constructing a glossary (84,85,104,105) for masking the noise features utilizing at least the discrepancy list.
A system for determining whether at least one document A is a candidate for near duplicate to a document B;
- at least two of said documents include noise features being nominally identical features that appear differently in the documents, the system comprising:
- a glossary (85,105);
- configured to map noise features to common mapped features;
- a repository (83,106) storing documents A;
- near-duplicate module (81,103) configured to process said document B and at least one of said stored documents A, for determining at least one document A that is candidate for near duplicate to document B;
- a system (86, 102, 87 107) configured to be applied to for each one of at least one of said candidate documents A, and capable of
- (i) utilizing the glossary, for said candidate document A and document B, for masking noise features, if any, by mapping said noise features into common mapped features, giving rise to masked documents;
- (ii) determine resemblance between said masked documents.
A system for determining whether at least one masked document A is a candidate for near duplicate to a document B;
- at least two of said documents include noise features being nominally identical features that appear differently in the documents;
- the system comprising:
- a glossary (85,105) configured to map noise features to common mapped features;
- a repository (83,106) storing masked documents A;
- a system (86, 102) configured to utilize the glossary, for said document B, for masking noise features, if any, by mapping said noise features into common mapped features, giving rise to masked document B;
- a near-duplicate module (81,103) to the masked documents, for determining at least one masked document A that is candidate for near duplicate to the masked document B;
- a system (86, 102, 87 107,84,104) configured to be applied to for each one of at least one of said candidate documents A, and capable of
- (i) utilizing the glossary, for said candidate document A and masked document B, for masking noise features, if any, by mapping said noise features into common mapped features, giving rise to masked documents;
- (ii) determine resemblance between said masked documents;
- and above threshold,
- (iii) identifying, in said document and masked document B, at least one pair of noise features, if any, which do not appear in the glossary, and in respect of each pair, mapping said pair of features to a respective common mapped feature and automatically adding said mapped feature to the glossary.
Dependencies are inferred from the claim text.