System and method for textual near-duplicate grouping of documents

A system for efficiently grouping electronic documents that are likely textual near-duplicates, the system comprising:

• a first memory storing an electronic document repository containing a plurality of electronic documents, including a first electronic document and a second electronic document;
• a computing device comprising one or more processors and a second memory, the second memory storing instructions that, when executed by the one or more processors, cause the computing device to obtain electronic documents stored in the first memory to
• process the first electronic document and the second electronic document to determine respective sets of character sequence counts, at least in part by, for each electronic document,
• (a) identifying a plurality of non-contiguous character sequences expressed within text of the electronic document, each non-contiguous character sequence corresponding to a different starting position within the text and including (i) a first character at the respective starting position, and (ii) a second character at a pre-defined offset from the respective starting position, and
• (b) determining a set of character sequence counts, at least in part by determining, for each unique non-contiguous character sequence occurring within the identified plurality of non-contiguous character sequences, a character sequence count specifying how many times the unique non-contiguous character sequence is expressed within the text of the electronic document,
• generate one or more similarity metrics, at least in part by comparing (i) the set of character sequence counts determined for the first electronic document to (ii) the set of character sequence counts determined for the second electronic document, and
• assign, based on the one or more similarity metrics, the second electronic document to a same document group as the first electronic document.

A non-transitory, computer-readable medium storing instructions that, when executed by one or more processors of a computing device, cause the computing device to:

• process a first electronic document and a second electronic document to determine respective sets of character sequence counts, at least in part by, for each electronic document,
• (a) identifying a plurality of non-contiguous character sequences expressed within text of the electronic document, each non-contiguous character sequence corresponding to a different starting position within the text and including (i) a first character at the respective starting position, and (ii) a second character at a pre-defined offset from the respective starting position, and
• (b) determining a set of character sequence counts, at least in part by determining, for each unique non-contiguous character sequence occurring within the identified plurality of non-contiguous character sequences, a character sequence count specifying how many times the unique non-contiguous character sequence is expressed within the text of the electronic document;
• generate one or more similarity metrics, at least in part by comparing (i) the set of character sequence counts determined for the first electronic document to (ii) the set of character sequence counts determined for the second electronic document;
• and
• assign, based on the one or more similarity metrics, the second electronic document to a same document group as the first electronic document.