Detecting duplicate tables in data lake databases
AssigneeInternational Business Machines Corporation
InventorsKarina Elayne Kervin, Jian Wu, SIBASIS DAS, Radha Mohan De, Swaminathan Balasubramanian, CHERANELLORE VASUDEVAN
Life of this patent
Estimating the term…
Abstract
Detecting duplicate tables by converting relational tables to knowledge graphs; and mapping nodes in the knowledge graphs to sources in the relational tables. The method may further include applying graph matching to the knowledge graphs; and assessing degree of matching between matched knowledge graphs.
Claims20
A computer implemented method of detecting duplicate tables comprising:
- converting syntactical data from relational tables from a schema of a database to knowledge graphs having semantic data;
- mapping nodes in the knowledge graphs to sources in the relational tables;
- applying graph matching to the knowledge graphs;
- detecting semantically equivalent tables from the relational tables by assessing a degree of matching between matched knowledge graphs based on equivalent data from common linked-pair nodes, defined from relationships, from the knowledge graphs;
- and
- generating a corrective flag to eliminate the semantically equivalent tables from the relational tables from the schema of the database based on the degree of matching.
A system for detecting duplicate tables including a hardware processor;
- and a memory that stores a computer program product, the computer program product of the system includes instructions comprising:
- convert, using a hardware processor, syntactical data from relational tables from a schema of a database to knowledge graphs having semantic data;
- map, using a hardware processor, nodes in the knowledge graphs to sources in the relational tables;
- apply, using a hardware processor, graph matching to the knowledge graphs;
- and
- detect, using the hardware processor, semantically equivalent tables from the relational tables by assessing a degree of matching between matched knowledge graphs based on equivalent data from common linked-pair nodes, defined from relationships, from the knowledge graphs;
- and
- generate, using the hardware processor, a corrective flag to eliminate the semantically equivalent tables from the relational tables from the schema of the database based on the degree of matching.
A computer program product for detecting duplicate tables, the computer program product including a computer readable storage medium having computer readable program code embodied therewith, the program instructions executable by a processor to cause the processor to:
- convert syntactical data from relational tables from a schema of a database to knowledge graphs having semantic data;
- map nodes in the knowledge graphs to sources in the relational tables;
- apply graph matching to the knowledge graphs;
- detect semantically equivalent tables from the relational tables by assessing a degree of matching between matched knowledge graphs based on equivalent data from common linked-pair nodes, defined from relationships, from the knowledge graphs;
- and
- generate a corrective flag to eliminate the semantically equivalent tables from the relational tables from the schema of the database based on the degree of matching.
Dependencies are inferred from the claim text.