# US 2014/0156606 A1 · Method and System for Integrating Data Into a Database > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2014/0156606 A1 - Document ID: US20140156606A1 - Status: Published application - Kind: A1 - Application number: 13/928,983 - Filed: 2013-06-27 - Published: 2014-06-05 - Earliest claimed: 2012-07-16 - Assignees: Qatar Foundation - Applicants: Qatar Foundation - Inventors: George BESKALES; Ihab Francis IIyas KALDAS - Web page: https://patentgrep.com/patent/US20140156606A1 ## CPC - G06F 16/25: Integrating or interfacing systems involving database management systems - G06F 16/215: Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract A method for integrating data into a database comprises storing data comprising a plurality of records which each comprise a plurality of attributes; analysing a sample of records from the plurality of records by: identifying duplicate pairs of records in the sample records; analysing each attribute of each record of the duplicate pairs of records to identify a respective attribute condition which is indicative that the pairs of records are duplicates; wherein the method further comprises: comparing each attribute of a record with the respective attribute condition and, if the attribute satisfies the attribute condition, allocating the record to a disjoint group which comprises records with an attribute that satisfies the same respective attribute condition; identifying duplicate pairs of records in the records in each disjoint group; identifying duplicate pairs of records in records that are not allocated to a disjoint group; and consolidating each duplicate pair of records into one consolidated record and storing the consolidated record in an integrated database. ## Claims ### Claim 1 A method for integrating data into a database, the method comprising: storing data comprising a plurality of records which each comprise a plurality of attributes; analysing a sample of records from the plurality of records by: identifying duplicate pairs of records in the sample records; and analysing each attribute of each record of the duplicate pairs of records to identify a respective attribute condition which is indicative that the pairs of records are duplicates; comparing each attribute of a record with the respective attribute condition and, if the attribute satisfies the attribute condition, allocating the record to a disjoint group which comprises records with an attribute that satisfies the same respective attribute condition; identifying duplicate pairs of records in the records in each disjoint group; identifying duplicate pairs of records in records that are not allocated to a disjoint group; and consolidating each duplicate pair of records into one consolidated record and storing the consolidated record in an integrated database. ### Claim 2 (depends on 1) A method according to claim 1, wherein records are allocated to each disjoint group in linear time. ### Claim 3 (depends on 1) A method according to claim 1, wherein the method further comprises truncating predetermined words which are insignificant for deduplication from attributes of records. ### Claim 4 (depends on 1) A method according to claim 1, wherein the method further comprises identifying distinctive features of records. ### Claim 5 (depends on 4) A method according to claim 4, wherein the method comprises allocating each record which comprises a distinctive feature to a respective disjoint group which comprises records with the same distinctive feature. ### Claim 6 (depends on 1) A method according to claim 1, wherein the method comprises only allocating a record to a disjoint group if, after a predetermined time, there are more than a predetermined number of records in the disjoint group. ### Claim 7 (depends on 1) A method according to claim 1, wherein identifying duplicate pairs of records comprises: analysing the attributes of the records; identifying candidate pairs of records, each candidate pair comprising a two records which have at least one similar attribute; generating a similarity value which is indicative of the level of similarity between the two records in each candidate pair; identifying each candidate pair with a similarity value above a predetermined level as a duplicate pair of records. ### Claim 8 (depends on 7) A method according to claim 7, wherein, before the step of analysing the attributes of the records, the method further comprises learning at least one rule using sample data and then subsequently using each rule in the method to identify the candidate pairs of records and generate the similarity value. ### Claim 9 (depends on 8) A method according to claim 8, wherein the sample data comprises only duplicate records. ### Claim 10 (depends on 8) A method according to claim 8, wherein the method comprises grouping the records in the sample data into groups of records with records in each group having at least one attribute which is the same as at least one attribute in another record in the group. ### Claim 11 (depends on 7) A method according to claim 7, wherein the similarity value is a conditional probability value which is indicative of the probability of the two records in a candidate pair being duplicates of one another. ### Claim 12 (depends on 11) A method according to claim 11, wherein the step of learning each rule comprises calculating the probability distribution by comparing the attributes of known duplicate records with a random sample of pairs of records. ### Claim 13 (depends on 12) A method according to claim 12, wherein the calculation is based on the Bayes' rule. ### Claim 14 (depends on 12) A method according to claim 12, wherein the method further comprises calculating the conditional probability value from the probability distributions using the Naïve Bayes' rule. ### Claim 15 (depends on 7) A method according to claim 7, wherein the method further comprises comparing the similarity value of each candidate pair with the similarity value of each other candidate pair and clustering the records into a set of disjoint clusters. ### Claim 16 (depends on 15) A method according to claim 15, wherein the method comprises storing information representing each cluster for use with further data. ### Claim 17 (depends on 1) A method according to claim 1, wherein the method comprises using a plurality of rules to identify duplicate records. ### Claim 18 (depends on 1) A method according to claim 1, wherein the steps of the method are repeated for a plurality of data sources, one data source at a time. ### Claim 19 A non-transitory computer-readable media embedded on a non-transitory tangible computer readable storage medium, the non-transitory computer-readable media including machine readable instructions that, when executed by a processor, implement a computer-implemented method for integrating data into a database, the method comprising: storing data comprising a plurality of records which each comprise a plurality of attributes; analysing a sample of records from the plurality of records by: identifying duplicate pairs of records in the sample records; and analysing each attribute of each record of the duplicate pairs of records to identify a respective attribute condition which is indicative that the pairs of records are duplicates; comparing each attribute of a record with the respective attribute condition and, if the attribute satisfies the attribute condition, allocating the record to a disjoint group which comprises records with an attribute that satisfies the same respective attribute condition; identifying duplicate pairs of records in the records in each disjoint group; identifying duplicate pairs of records in records that are not allocated to a disjoint group; and consolidating each duplicate pair of records into one consolidated record and storing the consolidated record in an integrated database.