Method and apparatus for detecting and summarizing document similarity within large document sets

A system for comparing a query file to one or more stored files, the system comprising:

• a file segmenter for creating a plurality of query file substrings from a query file;
• a substring preprocessor for preprocessing the first query file substring thereby making the substring more suitable for searching in the storage area:
• a storage searcher for searching a storage area storing a plurality of ordered file substrings for a first query file substring;
• a data storer for storing match data relating to a match between the first query file substring and a first ordered file substring;
• and
• a substring coalescer for joining the first ordered file substring and a second ordered file substring if the first ordered file substring and the second ordered file substring are in a particular sequence and for joining the first query file substring and a second query file substring if the first query file substring and the second query file substring are in the same particular sequence wherein the second ordered file substring and the second query file substring match, the substring coalescer thereby forming a third coalesced ordered file substring and a third coalesced query file substring that can be used to format output comparison data.

A computer readable medium containing programmed instructions for comparing a query file to one or more stored files, the programmed instructions comprising:

• a computer code for receiving a query file having a plurality of query file substrings;
• a computer code for selecting a first query file substring from the plurality of query file sub strings;
• a computer code for preprocessing the first query file substring thereby making the substring more suitable for searching in the storage area
• a computer code for searching a storage area storing a plurality of ordered file substrings for the first query file substring;
• a computer code for storing match data relating to a match between the first query file substring and a first ordered file substring;
• and
• a computer code for joining the first ordered file substring and a second ordered file substring if the first ordered file substring and the second ordered file substring are in a particular sequence and for joining the first query file substring and a second query file substring if the first query file substring and the second query file substring are in the same particular sequence wherein the second ordered file substring and the second query file substring match, thereby forming a third coalesced ordered file substring and a third coalesced query file substring that can be used to format output comparison data.