Proactive duplicate identification

A method for proactive duplicate identification, the method comprising:

• determining, by a hardware implemented document data extractor that is executed by at least one hardware processor, data related to a document;
• indexing, by a hardware implemented data indexer and analyzer that is executed by the at least one hardware processor, documents from at least one of history data and current data for duplicate analysis of the document, wherein
• a time-frame for the history data is greater than a time-frame for the current data,
• the indexing of the documents includes an initial indexing based on the determination of the data related to the document for an initial data field, and a further indexing based on a further determination of further data related to the document for a further data field, and
• the initial indexing based on the initial data field narrows a scope of the documents that are used from the at least one of the history data and the current data for the duplicate analysis of the document, and the further indexing based on the further data field further narrows the scope of the documents that are used from the at least one of the history data and the current data for the duplicate analysis of the document;
• utilizing, by the hardware implemented data indexer and analyzer, the indexed documents to perform the duplicate analysis of the document to determine whether the document is a duplicate document, a probable duplicate document, or a non-duplicate document;
• and
• in response to a determination that the document is a non-duplicate document, forwarding, by the hardware implemented data indexer and analyzer, the document to a target database for further analysis.

A non-transitory computer readable medium having stored thereon machine readable instructions for proactive duplicate identification, the machine readable instructions when executed cause at least one hardware processor to:

• determine, by a hardware implemented document data extractor that is executed by the at least one hardware processor, data related to a document;
• generate, by a hardware implemented data indexer and analyzer that is executed by the at least one hardware processor, a real-time initial index of documents from at least one of history data and current data for duplicate analysis of the document based on the determination of the data related to the document for an initial data field, wherein a time-frame for the history data is greater than a time-frame for the current data;
• generate, by the hardware implemented data indexer and analyzer, a real-time further index of the documents from the at least one of the history data and the current data for duplicate analysis of the document based on a further determination of further data related to the document for a further data field, wherein the real-time initial index narrows a scope of the documents that are used for the duplicate analysis of the document, and the real-time further index further narrows the scope of the documents that are used for the duplicate analysis of the document;
• and
• utilize, by the hardware implemented data indexer and analyzer, real-time indexed documents from the real-time initial index and the real-time further index to perform the duplicate analysis of the document to determine whether the document is a duplicate document, a probable duplicate document, or a non-duplicate document.