Integrated fuzzy joins in database management systems

A device comprising:

• a processor;
• and
• executable instructions operable by the processor, the executable instructions comprising a method for generating fuzzy loins between records, the method comprising:
• tokenizing two records of different datasets into tokens;
• determining transforms for each token of the tokenized records;
• generating signatures for the two tokenized records using the transforms;
• executing a user-defined function to perform matching of signatures within the runtime layer of a database system;
• invoking an object manager by submission of the user-defined function, the object manager maintaining persistence of an object storing the tokens for each record and the associated transforms between a signature generation query and a similarity computation query;
• building signature tables with the generated signatures;
• identifying two signature tables with at least one common signature, wherein the two signature tables are related to the two records with different datasets;
• equi-joining the two signature tables based on their at least one common signature, wherein the equi-joining comprises joining record identifications from the two signature tables and combining other fields from the records in the two different datasets as identified by the joined record identifications;
• computing a similarity measure between the two records corresponding to the two signature tables;
• and
• creating a fuzzy join between the two records with the at least one common signature when the similarity measure is above a threshold.

A system, comprising:

• a memory storing a plurality of records;
• a tokenizer configured to tokenize each of the plurality of records of a first dataset and a second dataset into one or more tokens;
• a transformer configured to find transforms for each of the one or more tokens;
• a signature generator configured to run within a runtime layer of a database system to generate signatures for the plurality of records based on the one or more tokens and the transforms and to build signature tables with the generated signatures;
• a fuzzy index generator configured to build a fuzzy index for the first dataset and the second dataset with the corresponding signature and record identifier;
• an identifier to identify two signature tables with at least one common signature;
• a similarity calculator configured to run within the runtime layer of the database system to generate a measure of similarity between one of the plurality of records in the first dataset and one of the plurality of records in the second dataset, wherein the signature generator is further configured to invoke an object manager configured to maintain persistence of an object storing the tokens for each record and the associated transforms between a signature generation query performed by the signature generator and a similarity computation query performed by the similarity calculator;
• and
• a joiner configured to equi-join the two signature tables based on their at least one common signature and to create a fuzzy join between equi-joined records by assigning the similarity measure to the join, wherein the equi-joining comprises joining record identifications from the two signature tables and combining other fields from the records in the first dataset and the second dataset as identified by the joined record identifications.