Data de-duplication system
AssigneeOracle International Corporation
InventorsDonald Kacher, Kishore Srirambhatla, Nanda Venugopal, Abhishek Raizada, Abhay Kulkarni
Life of this patent
Estimating the term…
Abstract
A data de-duplication system is provided that supports the loading and integration of data from multiple data sources. The data de-duplication system identifies and merges duplicate dimension data records that describe the same entity by creating a single dimension data record that is identified as a single best record (“SBR”). The data de-duplication system further adjusts foreign keys that reference the duplicate dimension data records so that the foreign keys correctly reference the merged dimension data record (i.e., the SBR).
Claims17
A non-transitory computer-readable medium having instructions stored thereon that, when executed by a processor, cause the processor to manage duplicate data originating from a plurality of data sources, the managing of the duplicate data comprising:
- generating one or more de-duplication instructions for a data warehouse, wherein the one or more de-duplication instructions identify dimension data records that are considered duplicate dimension data records and identify a single dimension data record that represents the duplicate dimension data records;
- receiving dimension data from a plurality of data sources, wherein the dimension data comprises one or more dimension data records;
- identifying at least two duplicate dimension data records by applying the one or more de-duplication instructions to the received dimension data;
- replacing the at least two duplicate dimension data records with the single dimension data record within the data warehouse;
- adjusting one or more foreign keys of at least one data record within the data warehouse to reference the single dimension data record;
- receiving one or more additional de-duplication instructions, wherein the one or more additional de-duplication instructions identify dimension data records that are no longer considered duplicate dimension data records;
- replacing the single dimension data record with the at least two duplicate dimension data records within the data warehouse;
- and
- adjusting the one or more foreign keys of the at least one data record within the data warehouse to reference the at least two duplicate dimension data records.
A computer-implemented method for managing duplicate data originating from a plurality of data sources, the computer-implemented method comprising:
- generating one or more de-duplication instructions for a data warehouse, wherein the one or more de-duplication instructions identify dimension data records that are considered duplicate dimension data records and identify a single dimension data record that represents the duplicate dimension data records;
- receiving dimension data from a plurality of data sources, wherein the dimension data comprises one or more dimension data records;
- identifying at least two duplicate dimension data records by applying the one or more de-duplication instructions to the received dimension data;
- replacing the at least two duplicate dimension data records with the single dimension data record within the data warehouse;
- adjusting one or more foreign keys of at least one data record within the data warehouse to reference the single dimension data record;
- receiving one or more additional de-duplication instructions, wherein the one or more additional de-duplication instructions identify dimension data records that are no longer considered duplicate dimension data records;
- replacing the single dimension data record with the at least two duplicate dimension data records within the data warehouse;
- and
- adjusting the one or more foreign keys of the at least one data record within the data warehouse to reference the at least two duplicate dimension data records.
A data de-duplication system, comprising:
- a processor;
- a generating module configured to generate one or more de-duplication instructions for a data warehouse, wherein the one or more de-duplication instructions identify dimension data records that are considered duplicate dimension data records and identify a single dimension data record that represents the duplicate dimension data records;
- a receiving module configured to receive dimension data from a plurality of data sources, wherein the dimension data comprises one or more dimension data records;
- an identifying module configured to identify at least two duplicate dimension data records by applying the one or more de-duplication instructions to the received dimension data;
- a replacing module configured to replace the at least two duplicate dimension data records with the single dimension data record within the data warehouse;
- and
- an adjusting module configured to adjust one or more foreign keys of at least one data record within the data warehouse to reference the single dimension data record;
- wherein the receiving module is further configured to receive one or more additional de-duplication instructions, wherein the one or more additional de-duplication instructions identify dimension data records that are no longer considered duplicate dimension data records;
- wherein the replacing module is further configured to replace the single dimension data record with the at least two duplicate dimension data records within the data warehouse;
- and
- wherein the adjusting module is further configured to adjust the one or more foreign keys of the at least one data record within the data warehouse to reference the at least two duplicate dimension data records.
Dependencies are inferred from the claim text.