Data cleaning
AssigneeQatar Foundation
InventorsIhab Francis Ilyas Kaldas, George Beskales, Ahmed Elmagarmid
Life of this patent
Estimating the term…
Abstract
A computer-implemented method comprising partitioning data representing an input instance of a database including multiple tuples into multiple fragments of tuples, detecting tuples which violate a data quality specification in respective ones of the fragments, selecting a data cleaning asset on the basis of characteristics of errors in detected tuples for a fragment and based on declared asset capabilities, assigning a selected data cleaning asset to the fragment, the selected data cleaning asset to provide a set of candidate corrections for the detected tuples in the fragment, providing data representing an output instance of the database in which detected tuples are replaced with selected candidate corrections.
Claims12
A computer-implemented method comprising:
- partitioning data representing an input instance of a database including multiple tuples into multiple fragments of tuples;
- detecting tuples which violate a data quality specification in respective ones of the fragments;
- selecting multiple data cleaning assets on the basis of characteristics of errors in detected tuples for a fragment and based on declared asset capabilities;
- assigning multiple selected data cleaning assets to the fragment, the selected multiple data cleaning assets to provide sets of redundant candidate corrections for the detected tuples in the fragment;
- selecting a candidate correction for a tuple with a relatively higher confidence measure from measures for the candidate corrections in the redundant sets, and wherein a confidence measure includes a measure representing a majority vote for a tuple from multiple candidate corrections for the tuple from the redundant sets;
- and
- providing data representing an output instance of the database in which detected tuples are replaced with selected candidate corrections.
A system comprising:
- a processor to process a database including multiple data records to be cleaned;
- multiple data cleaning assets to determine a set of data records of the database which violate at least one data quality description;
- and
- a selection and allocation module to select a data cleaning asset from a set of assets including a user from a crowdsourcing system and a machine-based data cleaning asset, to allocate a selected asset to a fragment of the data records in the set, the selected asset to generate a candidate correction for a data record in the set, and to select candidate corrections by determining a clean database instance that includes the maximum number tuples which obey the data quality description;
- and using a confidence measure which includes a measure representing a majority vote for a tuple from multiple candidate corrections for the tuple from redundant sets of corrections.
A computer program embedded on a non-transitory tangible computer readable storage medium, the computer program including machine readable instructions that, when executed by a processor, implement a method comprising:
- partitioning data representing an input instance of a database including multiple tuples into multiple fragments of tuples;
- detecting tuples which violate a data quality specification in respective ones of the fragments;
- selecting a data cleaning asset on the basis of characteristics of errors in detected tuples for a fragment and based on declared asset capabilities;
- assigning multiple selected data cleaning assets to the fragment, the selected multiple data cleaning assets to provide sets of redundant candidate corrections for the detected tuples in the fragment;
- selecting a candidate correction for a tuple with a relatively higher confidence measure from measures for the candidate corrections in the redundant sets, and wherein a confidence measure includes a measure representing a majority vote for a tuple from multiple candidate corrections for the tuple from the redundant sets;
- and
- providing data representing an output instance of the database in which detected tuples are replaced with selected candidate corrections.
Dependencies are inferred from the claim text.