System for similar document detection

A method for detecting similar documents comprising:

• obtaining a document;
• filtering the document to eliminate tokens and obtain a filtered document containing remaining tokens, the tokens being eliminated based on grammatical components;
• sorting the filtered document to reorder the tokens according to a predetermined ranking;
• generating a single tuple for the filtered document;
• comparing the tuple for the filtered document with a document storage structure comprising a plurality of tuples, each tuple in the plurality of tuples representing one of a plurality of documents;
• and
• determining if the tuple for the filtered document is clustered with another tuple in the document storage structure, thereby detecting if the document is similar to another document represented by the another tuple in the document storage structure.

An apparatus for detecting similar documents comprising:

• means for obtaining a document;
• means for parsing the document to remove formatting and to obtain a token stream, the token stream comprising a plurality of tokens;
• means for retaining only retained tokens in the token stream by using at least one token threshold;
• means for reordering the retained tokens to obtain an arranged token stream;
• means for processing in turn each retained token in the arranged token stream using a hash algorithm to obtain a single hash value for the document;
• means for generating a document identifier for the document;
• means for forming a single tuple for the document, the tuple comprising the document identifier for the document and the hash value for the document;
• means for inserting the tuple for the document into a document storage tree, the document storage tree comprising a plurality of tuples, each tuple located at a bucket of the document storage tree, each tuple in the plurality of tuples representing one of a plurality of documents, each tuple in the plurality of tuples comprising a document identifier and a hash value;
• and
• means for determining if the tuple for the document is co-located with another tuple at a same bucket in the document storage tree, thereby detecting if the document is similar to another document represented by the another tuple in the document storage tree.