Detecting duplicate documents using classification

A computer program product, the computer program product comprising a computer usable medium having computer usable program code for managing a collection of documents, the code being configured for:

• receiving a document;
• determining, by operation of the one or more computer processors, a similarity score between the received document and each of a plurality of categories, wherein each category is assigned one or more documents;
• determining, based on the plurality of similarity scores, whether the received document is one of a duplicate and a near-duplicate of one of the documents assigned to one of the plurality of categories;
• upon determining that the received document is not a duplicate, or near duplicate, to a document assigned to any of the plurality of categories:
• creating a new category for the received document, and
• training a classifier associated with the new category using the received document, wherein the trained classifier is configured to determine a measure of similarity between the received document and an input document;
• and
• upon determining that the received document is a duplicate, or near duplicate, to a document assigned to at least one of the plurality of categories:
• assigning the document to the determined category, and
• training a classifier associated with the determined category using the received document.

A system, comprising:

• a processor;
• and
• a memory containing an application program configured to manage a collection of documents, which, when executed on the processor is configured to perform an operation comprising:
• receiving a document,
• determining, by operation of the one or more computer processors, a similarity score between the received document and each of a plurality of categories, wherein each category is assigned one or more documents,
• determining, based on the plurality of similarity scores, whether the received document is one of a duplicate and a near-duplicate of one of the documents assigned to one of the plurality of categories,
• upon determining that the received document is not a duplicate, or near duplicate, to a document assigned to any of the plurality of categories:
• creating a new category for the received document;
• and
• training a classifier associated with the new category using the received document, wherein the trained classifier is configured to determine a measure of similarity between the received document and an input document, and
• upon determining that the received document is a duplicate, or near duplicate, to a document assigned to at least one of the plurality of categories:
• assigning the document to the determined category;
• and
• training a classifier associated with the determined category using the received document.