Systems and methods for identifying duplicate documents and detecting misrepresentation

A method comprising:

• retrieving, via a network, from a first source from a first source comprising a repository of previously submitted documents, a first document;
• retrieving, via the network, from a second source distinct from the first source and comprising a repository of newly submitted documents, a second document;
• processing the first document and the second document, wherein processing comprises:
• cleaning each of the first document and the second document by removing personally identifiable metadata and hazardous hidden content comprising malicious scripts or malware,
• tokenizing each of the first document and the second document by substituting sensitive data elements with non-sensitive tokens,
• removing stop words, and
• vectorizing each of the first document and the second document into numeric representations comprising sentence-level vectors and word vectors;
• align sentences between the first document and the second document based on cosine similarity between the sentence-level vectors generated during vectorizing, and, within aligned sentence pairs:
• determine, based on the cosine similarity, a number W of words in common, and
• determine a number C of sentences in common equal to a count of aligned sentence pairs whose pairwise cosine similarity exceeds a sentence-pair threshold stored in the memory;
• determining a cosine similarity S between a first set of numbers and a second set of numbers, the first set of numbers corresponding to one or more sentences in the first document and the second set of numbers corresponding to one or more sentences in the second document, wherein each number in the first set of numbers corresponds to a word in the first document and each number in the second set of numbers corresponds to a word in the second document;
• determining whether the cosine similarity exceeds a first threshold stored in the memory;
• determining, based on the cosine similarity and the sentence alignment, a number of words in common between the first document and the second document equal to W;
• determining whether the number of words W in common exceeds a second threshold stored in the memory;
• determining, based on the cosine similarity and the sentence alignment, a number of sentences in common between the first document and the second document equal to C;
• determining whether the number of sentences in common C exceeds a third threshold stored in the memory;
• and
• responsive to a determination that the cosine similarity exceeds S the first threshold, and at least one of:
• the number of words in common W exceeds the second threshold, or the number of sentences in common C exceeds the third threshold:
• persistently write, to a record associated with the second document in a vector repository, a duplicate flag field stored as at least one bit that indicates that the second document is a duplicate.

A non-transitory computer readable medium having stored instructions, which when executed, cause at least one processor to perform instructions comprising:

• retrieving, via a network, from a first source from a first source comprising a repository of previously submitted documents, a first document;
• retrieving, via the network, from a second source distinct from the first source and comprising a repository of newly submitted documents, a second document;
• processing the first document and the second document, wherein processing comprises:
• cleaning each of the first document and the second document by removing personally identifiable metadata and hazardous hidden content comprising malicious scripts or malware,
• tokenizing each of the first document and the second document by substituting sensitive data elements with non-sensitive tokens, removing stop words, and
• vectorizing each of the first document and the second document into numeric representations comprising sentence-level vectors and word vectors;
• align sentences between the first document and the second document based on cosine similarity between the sentence-level vectors generated during vectorizing, and, within aligned sentence pairs:
• determine, based on the cosine similarity, a number W of words in common, and
• determine a number C of sentences in common equal to a count of aligned sentence pairs whose pairwise cosine similarity exceeds a sentence-pair threshold stored in the memory;
• determining a cosine similarity S between a first set of numbers and a second set of numbers, the first set of numbers corresponding to one or more sentences in the first document and the second set of numbers corresponding to one or more sentences in the second document, wherein each number in the first set of numbers corresponds to a word in the first document and each number in the second set of numbers corresponds to a word in the second document;
• determining whether the cosine similarity exceeds a first threshold stored in the memory;
• determining, based on the cosine similarity and the sentence alignment, a number of words in common between the first document and the second document equal to W;
• determining whether the number of words W in common exceeds a second threshold stored in the memory;
• determining, based on the cosine similarity and the sentence alignment, a number of sentences in common between the first document and the second document equal to C;
• determining whether the number of sentences in common C exceeds a third threshold stored in the memory;
• and
• responsive to a determination that the cosine similarity exceeds S the first threshold, and at least one of:
• the number of words in common W exceeds the second threshold, or the number of sentences in common C exceeds the third threshold:
• persistently write, to a record associated with the second document in a vector repository, a duplicate flag field stored as at least one bit that indicates that the second document is a duplicate.