Data de-duplication systems and methods

A supervised machine learning de-duplication system, comprising:

• a communications interface configured to receive data from a plurality of sources;
• and
• processing circuitry configured to
• compare one or more attributes of at least two sets of data to identify whether the at least two sets of data are duplicates,
• characterize as a binary indication whether the comparing indicates the one or more attributes of the at least two sets of data are a match and store in a memory the binary indication
• display on a display a representation of the binary indication and receive via a graphical user interface (GUI) a user-identified indication whether the at least two sets of data are duplicates,
• train a classifier with feedback from the user-identified indication, the classifier being a computer-implemented classifier,
• record in the memory the binary indication whether the at least two sets of data are duplicates and when the at least two sets of data are duplicates, remove at least one of the at least two sets of data so as to de-duplicate a portion of the at least two sets of data,
• run the classifier with a processor as an intelligent proxy for subsequent de-duplication decisions dividing data results of the comparing with other comparing steps to form a training set of data and a testing set of data;
• apply the training set of data to fit the classifier via a k-fold cross-validation process;
• test an accuracy of the classifier via the testing set of data;
• and
• fit a plurality of decision trees for an associated plurality of sub-samples of the data results.

A non-transitory computer-readable storage medium having stored thereon computer-readable instructions that when executed by circuitry performs a method of de-duplicating data, the method having steps comprising:

• receiving data through a communications interface from a plurality of sources;
• comparing one or more attributes of at least two sets of data to identify whether the at least two sets of data are duplicates;
• characterizing as a binary indication whether the comparing indicates the one or more attributes of the at least two sets of data are a match and storing in a memory the binary indication;
• displaying on a display a representation of the binary indication and receiving via a graphical user interface (GUI) a user-identified indication whether the at least two sets of data are duplicates;
• training a classifier with feedback from the user-identified indication, the classifier being a computer-implemented classifier;
• recording in the memory the binary indication whether the at least two sets of data are duplicates and when the at least two sets of data are duplicates, removing at least one of the at least two sets of data so as to de-duplicate a portion of the at least two sets of data;
• and
• running the classifier with a processor as an intelligent proxy for subsequent de-duplication decisions, wherein the binary indication includes one or more of a true positive indicator, a true negative indicator, a false positive, and a false negative indicator, and an accuracy of the classifier is a function of a total sum of the true positive indicators and the true negative indicators, divided by a total sum of the true positive indicators, the false negative indicators, the false positive indicators and the true negative indicators.