Duplicate entry detection system and method

A networked computing system providing an electronic marketplace for offering products for sale from a plurality of providers and detecting duplicate product description documents from the plurality of providers, the networked computing system comprising:

• a processor;
• and
• a memory;
• wherein the networked computing system is configured to, upon receiving a first document:
• generate a set of tokens for the first document, each token in the set of tokens representing a series of characters found in the first document;
• select one or more tokens of the set of tokens having an occurrence frequency in the first document less than a threshold frequency;
• execute a non-fielded relevance search on a token index using the selected one or more tokens, the token index comprising an index of tokens corresponding to strings in documents of a document corpus, wherein the non-fielded relevance search returns candidate documents from the document corpus with scores for each candidate document;
• for each candidate document with a relevance score above a scoring threshold, perform filtering on the candidate document and disqualify the candidate document if the candidate document is a mutually exclusive variation of the first document, wherein the mutually exclusive variation indicates that, while otherwise similar to the first document, the candidate document is not a duplicate of the first document;
• and
• provide a set of duplicate documents in which each duplicate document has a score above the scoring threshold and has not been disqualified.

A networked computing system providing an electronic marketplace for offering products for sale from a plurality of providers and detecting duplicate product description documents from the plurality of providers, the networked computing system comprising:

• a processor;
• and
• a memory;
• wherein the networked computing system is configured to, with regard to a first document:
• generate tokens for the first document, each generated token representing a series of characters found in the first document;
• execute a non-fielded relevance search on a token index, the token index comprising an index of tokens corresponding to strings in documents of a document corpus, and wherein the relevance search returns candidate documents with corresponding scores for each candidate document;
• refine the scores for each candidate document to include matching of non-identical terms not considered as matches in the non-fielded relevance search;
• for each candidate document with a relevance score above a threshold:
• identify a set of tokens from the first document associated with a type of information contained in the subject matter of the first document;
• identify a corresponding set of tokens from the candidate document associated with a matching type of information contained in the subject matter of the second document;
• compare the set of tokens from the first document to the set of tokens from the candidate document;
• disqualify the candidate document if the comparison indicates the candidate document does not describe subject matter similar to the subject matter of the first document;
• perform a variation detection evaluation between the first document and the candidate document;
• and
• disqualify the candidate document if the candidate document is a mutually exclusive variation of the first document, wherein the mutually exclusive variation indicates that, while otherwise similar to the first document, the candidate document is not a duplicate of the first document;
• and
• provide a set of duplicate documents in which each duplicate document has a score above the threshold and has not been disqualified.

A tangible computer-readable medium bearing computer-executable instructions which, when executed on a computing device, cause the computing device to carry out operations comprising:

• generating tokens for the first document, each token representing a series of characters found in the first document;
• executing a non-fielded relevance search on a token index, the token index comprising an index of tokens corresponding to strings in documents of a document corpus, wherein the relevance search returns candidate documents with scores for each candidate document;
• refining the scores for each candidate document include matching of non-identical terms not considered as matches in the non-fielded relevance search;
• for each candidate document with a score above a threshold, performing filtering on the candidate document and disqualifying as a candidate document each candidate document not describing subject matter similar to subject matter of the first document, wherein the filtering comprises:
• performing a variation detection between the first document and a candidate document and disqualifying the candidate document if the candidate document is a mutually exclusive variation of the first document, wherein the mutually exclusive variation indicates that, while otherwise similar to the first document, the candidate document is not a duplicate of the first document;
• and
• performing domain specific comparisons between the first document and the candidate document by:
• identifying, from the first document, a first set of tokens associated with a field of the first document, the field of the first document associated with a type of information;
• identifying, from the second document, a second set of tokens associated with a field of the second document, the field of the second document associated with the type of information;
• and
• comparing the first set of tokens to the second set of tokens, and disqualifying the candidate document if the candidate document does not describe similar subject matter as to the first document according to the comparison;
• and
• providing a set of duplicate documents, wherein the set of duplicate documents includes candidate documents that have scores above the threshold, and that have not been disqualified.