Methods, systems, and computer readable media for utilizing machine learning for merging duplicate data records

A system comprising:

• at least one processor;
• a memory element;
• a data record consolidation (DRC) manager stored in the memory element and when executed by the at least one processor is configured for receiving a plurality of duplicate data records, constructing a plurality of collection level feature attributes from the duplicate data records, evaluating divergent patterns or variations of feature attribute values of the plurality of collection level feature attributes, extracting one or more samples from the plurality of collection level feature attributes, and receiving domain expert stimulus input corresponding to the one or more samples;
• and
• a merging machine learning (MML) model stored in the memory element and when executed by the at least one processor is configured for utilizing the merging machine learning model to generate a master data record corresponding to the duplicate data records or a rule to derive the master data record, wherein the merging machine learning model is trained by the DRC manager, using training data comprising the one or more samples and the stimulus input, to combine input duplicate data records from the plurality of duplicate data records to form the master data record or to produce the rule to derive the master data record, and wherein the merging machine learning module is configured to be updated using addition training data comprising variations between the input duplicate data records and the master data record;
• wherein the one or more samples comprises a representative subset selected from the plurality of collection level feature attributes having a majority of the divergent patterns or variations of the feature attribute values;
• wherein the plurality of collection level feature attributes is associated with a plurality of collection groups consolidated from the duplicate data records;
• wherein evaluating divergent patterns or variations of feature attribute values by assigning deviation scores to the collection groups of the duplicate data records, each of the deviation scores representing a number of divergent patterns or variations in the feature attribute values of one of the collection groups;
• and
• wherein extracting the one or more samples comprises selecting the representative subset from ones of the collection groups having deviation scores above a predetermined threshold.

A non-transitory computer readable medium having stored thereon executable instructions that when executed by at least one processor of a computer cause the computer to perform steps comprising:

• receiving a plurality of duplicate data records;
• constructing a plurality of collection level feature attributes from the duplicate data records, wherein the plurality of collection level feature attributes is associated with a plurality of collection groups consolidated from the duplicate data records;
• evaluating divergent patterns or variations of feature attribute values of the plurality of collection level feature attributes by assigning deviation scores to the collection groups of the duplicate data records, each of the deviation scores representing a number of divergent patterns or variations in the feature attribute values of one of the collection groups;
• extracting one or more samples from the plurality of collection level feature attributes, wherein the one or more samples comprises a representative subset selected from ones of the collection groups having deviation scores above a predetermined threshold;
• receiving domain expert stimulus input corresponding to the one or more samples;
• training a merging machine learning model, using training data comprising the one or more samples and the stimulus input, to combine input duplicate data records from the plurality of duplicate data records to form a master data record or to produce a rule to derive the master data record;
• utilizing the merging machine learning model to generate the master data record corresponding to the duplicate data records or the rule to derive the master data record;
• analyzing variations between the input duplicate data records and the master data record;
• and
• updating the merging machine learning model using addition training data comprising the variations between the input duplicate data records and the master data record.