Method and System for Integrating Data Into a Database

A non-transitory computer-readable media embedded on a non-transitory tangible computer readable storage medium, the non-transitory computer-readable media including machine readable instructions that, when executed by a processor, implement a computer-implemented method for integrating data into a database, the method comprising:

• storing data comprising a plurality of records which each comprise a plurality of attributes;
• analysing a sample of records from the plurality of records by:
• identifying duplicate pairs of records in the sample records;
• and
• analysing each attribute of each record of the duplicate pairs of records to identify a respective attribute condition which is indicative that the pairs of records are duplicates;
• comparing each attribute of a record with the respective attribute condition and, if the attribute satisfies the attribute condition, allocating the record to a disjoint group which comprises records with an attribute that satisfies the same respective attribute condition;
• identifying duplicate pairs of records in the records in each disjoint group;
• identifying duplicate pairs of records in records that are not allocated to a disjoint group;
• and
• consolidating each duplicate pair of records into one consolidated record and storing the consolidated record in an integrated database.