Method and apparatus for loading data files into a data-warehouse system

A method of loading data into a data-warehouse system for use in a data-warehouse system wherein there is to be data loaded or restored for use, said system comprising at least one computing device communicatively coupled with a large dataset, said large dataset having at least one storage device, said at least one storage device containing a full-backup data file and at least one partial-backup data file, said data files containing real-world entity data, said computing device having a Extraction Load Program (ETL) program to subject the data intended to be loaded to a transform stage according to data-warehouse-specific rules, the method comprising the step of:

• loading a first partial-backup data file,
• wherein the loading of said first partial-backup data file makes the data in said first partial-backup data file available for use, at a time when no other backup data files have been loaded,
• wherein said first partial-backup data file is one partial-backup data file within a series of historical partial-backup files for said data-warehouse system, and
• wherein said first partial-backup data file is not a full-backup data file;
• thereby loading data into the data-warehousing system, further comprising the step of:
• loading a full-backup data file,
• wherein each record to be extracted from said full-backup data file for loading into said data-warehouse system is only allowed to be written to said data-warehouse system if either the record to be written to said data-warehouse system is not already represented in said data-warehouse system or the record to be written to said data-warehouse system is designated as newer than its corresponding existing record, further comprising the steps of:
• defining a data-containing unit for use in constructing, extracting, and validating each record to be written in said data-warehouse system from said full-backup file;
• defining a dependent query based on database constraints or based on extrapolated relationships between data fields,
• wherein said dependent query is designed to populate said data-containing unit;
• performing said dependent query for each record in said full-backup data file,
• wherein said data-containing unit is populated and pushed into said ETL program,
• wherein referential integrity is maintained for all extracted data that is pushed to said ETL program, and
• wherein additional queries are made for all related entity data and said related entity data is pushed into said ETL program;
• validating and transforming each said dependent-query data set with said ETL program;
• and
• pushing only each said validated and transformed data set into said data-warehouse system.

A method of loading data into a data-warehouse system for use in a data-warehouse system wherein there is to be data loaded or restored for use, said system comprising at least one computing device communicatively coupled with a large dataset, said large dataset having at least one storage device, said at least one storage device containing a full-backup data file and at least one partial-backup data file, said data files containing real-world entity data, said computing device having a Extraction Load Program (ETL) program to subject the data intended to be loaded to a transform stage according to data-warehouse-specific rules, the method comprising the step of:

• loading a first partial-backup data file,
• wherein the loading of said first partial-backup data file makes the data in said first partial-backup data file available for use, at a time when no other backup data files have been loaded,
• wherein said first partial-backup data file is one partial-backup data file within a series of historical partial-backup files for said data-warehouse system, and
• wherein said first partial-backup data file is not a full-backup data file;
• thereby loading data into the data-warehousing system, further comprising the step of:
• loading at least one additional partial-backup data file,
• wherein the loading of each said at least one additional partial-backup data file makes the data in each at least one additional partial-backup data file available for use, at a time when other backup data files are available, but have not been loaded,
• wherein said at least one additional partial-backup data file is any partial-backup data file within a series of historical partial-backup files for said data-warehouse system,
• wherein said at least one additional partial-backup data file is not a full-backup data file, and
• wherein each record to be extracted from said at least one additional partial-backup data file for loading into said data-warehouse system is only allowed to be written to said data-warehouse system if either the record to be written to said data-warehouse system is not already represented in said data-warehouse system or the record to be written to said data-warehouse system is designated as newer than its corresponding existing record,
• wherein the relative age of the record to be written to said data-warehouse system as compared to its corresponding existing record is determined by comparing the relative ordering indexes of the records,
• wherein said relative ordering indexes of the records employs a strategy that breaks a data refresh into a backward-fill migration and a forward-fill migration,
• wherein the relative ordering index value for said backward-fill migration is assigned a value that will always cause a comparison of said value with an actual date of another record to result in said actual date being evaluated as newer, thus allowing the associated record under consideration to be overwritten,
• wherein the first introduction of any record will result in the insertion of said record into said data-warehouse system,
• wherein each new record is assigned a relative ordering index value of the date that said record was pushed into said data-warehouse system,
• wherein during said backward-fill migration, any additional records corresponding to a record already loaded in said data-warehouse system will be ignored,
• wherein during said forward-fill migration, a newer record will always overwrite an existing corresponding record in said data-warehouse system.

A method of loading data into a data-warehouse system for use in a data-warehouse system wherein there is to be data loaded or restored for use, said system comprising at least one computing device communicatively coupled with a large dataset, said large dataset having at least one storage device, said at least one storage device containing a full-backup data file and at least one partial-backup data file, said data files containing real-world entity data, said computing device having a Extraction Load Program (ETL) program to subject the data intended to be loaded to a transform stage according to data-warehouse-specific rules, the method comprising the step of:

• loading a first partial-backup data file,
• wherein the loading of said first partial-backup data file makes the data in said first partial-backup data file available for use, at a time when no other backup data files have been loaded,
• wherein said first partial-backup data file is one partial-backup data file within a series of historical partial-backup files for said data-warehouse system, and
• wherein said first partial-backup data file is not a full-backup data file;
• thereby loading data into the data-warehousing system, further comprising the step of:
• loading a full-backup data file,
• wherein each record to be extracted from said full-backup data file for loading into said data-warehouse system is only allowed to be written to said data-warehouse system if either the record to be written to said data-warehouse system is not already represented in said data-warehouse system or the record to be written to said data-warehouse system is designated as newer than its corresponding existing record,
• wherein the relative age of the record to be written to said data-warehouse system as compared to its corresponding existing record is determined by comparing the relative ordering indexes of the records,
• wherein said relative ordering indexes of the records employs a strategy that breaks a data refresh into a backward-fill migration and a forward-fill migration,
• wherein the relative ordering index value for said backward-fill migration is assigned a value that will always cause a comparison of said value with an actual date of another record to result in said actual date being evaluated as newer, thus allowing the associated record under consideration to be overwritten,
• wherein the first introduction of any record will result in the insertion of said record into said data-warehouse system,
• wherein each new record is assigned a relative ordering index value of the date that said record was pushed into said data-warehouse system,
• wherein during said backward-fill migration, any additional records corresponding to a record already loaded in said data-warehouse system will be ignored,
• wherein during said forward-fill migration, a newer record will always overwrite an existing corresponding record in said data-warehouse system.

A method of loading data into a data-warehouse system for use in a data-warehouse system wherein there is to be data loaded or restored for use, said system comprising at least one computing device communicatively coupled with a large dataset, said large dataset having at least one storage device, said at least one storage device containing a full-backup data file and at least one partial-backup data file, said data files containing real-world entity data, said computing device having a Extraction Load Program (ETL) program to subject the data intended to be loaded to a transform stage according to data-warehouse-specific rules, the method comprising the step of:

• loading a first partial-backup data file,
• wherein the loading of said first partial-backup data file makes the data in said first partial-backup data file available for use, at a time when no other backup data files have been loaded,
• wherein said first partial-backup data file is one partial-backup data file within a series of historical partial-backup files for said data-warehouse system, and
• wherein said first partial-backup data file is not a full-backup data file;
• thereby loading data into the data-warehousing system, further comprising the step of:
• loading at least one additional partial-backup data file,
• wherein the loading of each said at least one additional partial-backup data file makes the data in each at least one additional partial-backup data file available for use, at a time when other backup data files are available, but have not been loaded,
• wherein said at least one additional partial-backup data file is any partial-backup data file within a series of historical partial-backup files for said data-warehouse system,
• wherein said at least one additional partial-backup data file is not a full-backup data file, and
• wherein each record to be extracted from said at least one additional partial-backup data file for loading into said data-warehouse system is only allowed to be written to said data-warehouse system if either the record to be written to said data-warehouse system is not already represented in said data-warehouse system or the record to be written to said data-warehouse system is designated as newer than its corresponding existing record, further comprising the step of:
• loading a full-backup data file,
• wherein each record to be extracted from said full-backup data file for loading into said data-warehouse system is only allowed to be written to said data-warehouse system if either the record to be written to said data-warehouse system is not already represented in said data-warehouse system or the record to be written to said data-warehouse system is designated as newer than its corresponding existing record,
• wherein the relative age of the record to be written to said data-warehouse system as compared to its corresponding existing record is determined by comparing the relative ordering indexes of the records,
• wherein said relative ordering indexes of the records employs a strategy that breaks a data refresh into a backward-fill migration and a forward-fill migration,
• wherein the relative ordering index value for said backward-fill migration is assigned a value that will always cause a comparison of said value with an actual date of another record to result in said actual date being evaluated as newer, thus allowing the associated record under consideration to be overwritten,
• wherein the first introduction of any record will result in the insertion of said record into said data-warehouse system,
• wherein each new record is assigned a relative ordering index value of the date that said record was pushed into said data-warehouse system,
• wherein during said backward-fill migration, any additional records corresponding to a record already loaded in said data-warehouse system will be ignored,
• wherein during said forward-fill migration, a newer record will always overwrite an existing corresponding record in said data-warehouse system.

A method of loading data into a data-warehouse system for use in a data-warehouse system wherein there is to be data loaded or restored for use, said system comprising at least one computing device communicatively coupled with a large dataset, said large dataset having at least one storage device, said at least one storage device containing a full-backup data file and at least one partial-backup data file, said data files containing real-world entity data, said computing device having a Extraction Load Program (ETL) program to subject the data intended to be loaded to a transform stage according to data-warehouse-specific rules, the method comprising the step of:

• loading a first partial-backup data file,
• wherein the loading of said first partial-backup data file makes the data in said first partial-backup data file available for use, at a time when no other backup data files have been loaded,
• wherein said first partial-backup data file is one partial-backup data file within a series of historical partial-backup files for said data-warehouse system, and
• wherein said first partial-backup data file is not a full-backup data file;
• thereby loading data into the data-warehousing system, further comprising the step of:
• loading at least one additional partial-backup data file,
• wherein the loading of each said at least one additional partial-backup data file makes the data in each at least one additional partial-backup data file available for use, at a time when other backup data files are available, but have not been loaded
• wherein said at least one additional partial-backup data file is any partial-backup data file within a series of historical partial-backup files for said data-warehouse system,
• wherein said at least one additional partial-backup data file is not a full-backup data file, and
• wherein each record to be extracted from said at least one additional partial-backup data file for loading into said data-warehouse system is only allowed to be written to said data-warehouse system if either the record to be written to said data-warehouse system is not already represented in said data-warehouse system or the record to be written to said data-warehouse system is designated as newer than its corresponding existing record, further comprising the step of:
• loading a full-backup data file,
• wherein each record to be extracted from said full-backup data file for loading into said data-warehouse system is only allowed to be written to said data-warehouse system if either the record to be written to said data-warehouse system is not already represented in said data-warehouse system or the record to be written to said data-warehouse system is designated as newer than its corresponding existing record,
• wherein the relative age of the record to be written to said data-warehouse system as compared to its corresponding existing record is determined by comparing the relative ordering indexes of the records,
• further comprising the step of having a human user select said first partial-backup data file to be subjected to said backward-fill migration from other available backup data files, when any candidate first partial-backup files do not contain a date field that indicates the relative newness of the data as compared to other available backup data files, by evaluating the physical characteristics and known circumstances surrounding the creation of the backup data file under consideration to be said first partial-backup data file in order to assume a relative age of said selected first partial-backup data file for the purpose of implementing a strategy for employing relative ordering indexes for record loading during said backward-fill migration.

A data-warehouse system wherein there is to be data loaded or restored for use, said system having at least one computing device communicatively coupled with a large dataset, said large dataset having at least one storage device, said at least one storage device containing a full-backup data file and at least one partial-backup data file, said data files containing real-world entity data, said computing device having a Extraction Load Program (ETL) program to subject the data intended to be loaded to a transform stage according to data-warehouse-specific rules, comprising:

• a first partial-backup data file,
• wherein the loading of said first partial-backup data file makes the data in said first partial-backup data file available for use, at a time when no other backup data files have been loaded,
• wherein said first partial-backup data file is one partial-backup data file within a series of historical partial-backup files for said data-warehouse system, and
• wherein said first partial-backup data file is not a full-backup data file, further comprising:
• at least one additional partial-backup data file,
• wherein the loading of each said at least one additional partial-backup data file makes the data in each at least one additional partial-backup data file available for use, at a time when other backup data files are available, but have not been loaded,
• wherein said at least one additional partial-backup data file is any partial-backup data file within a series of historical partial-backup files for said data-warehouse system,
• wherein said at least one additional partial-backup data file is not a full-backup data file, and
• wherein each record to be extracted from said at least one additional partial-backup data file for loading into said data-warehouse system is only allowed to be written to said data-warehouse system if either the record to be written to said data-warehouse system is not already represented in said data-warehouse system or the record to be written to said data-warehouse system is designated as newer than its corresponding existing record, further comprising:
• a defined data-containing unit for use in constructing, extracting, and validating each record to be written in said data-warehouse system from said at least one additional partial-backup file;
• a defined dependent query based on database constraints or based on extrapolated relationships between data fields,
• wherein said dependent query is designed to populate said data-containing unit;
• a program to perform said dependent query for each record in said at least one additional partial-backup data file,
• wherein said data-containing unit is populated and pushed into said ETL program,
• wherein referential integrity is maintained for all extracted data that is pushed to said ETL program,
• wherein additional queries are made for all related entity data and said related entity data is pushed into said ETL program,
• wherein each said dependent-query data set is validated and transformed with said ETL program, and
• wherein only each said validated and transformed data set is pushed into said data-warehouse system.

A data-warehouse system wherein there is to be data loaded or restored for use, said system having at least one computing device communicatively coupled with a large dataset, said large dataset having at least one storage device, said at least one storage device containing a full-backup data file and at least one partial-backup data file, said data files containing real-world entity data, said computing device having a Extraction Load Program (ETL) program to subject the data intended to be loaded to a transform stage according to data-warehouse-specific rules, comprising:

• a first partial-backup data file,
• wherein the loading of said first partial-backup data file makes the data in said first partial-backup data file available for use, at a time when no other backup data files have been loaded,
• wherein said first partial-backup data file is one partial-backup data file within a series of historical partial-backup files for said data-warehouse system, and
• wherein said first partial-backup data file is not a full-backup data file, further comprising:
• a full-backup data file,
• wherein each record to be extracted from said full-backup data file for loading into said data-warehouse system is only allowed to be written to said data-warehouse system if either the record to be written to said data-warehouse system is not already represented in said data-warehouse system or the record to be written to said data-warehouse system is designated as newer than its corresponding existing record,
• further comprising:
• a defined data-containing unit for use in constructing, extracting, and validating each record to be written in said data-warehouse system from said full-backup file;
• a defined dependent query based on database constraints or based on extrapolated relationships between data fields,
• wherein said dependent query is designed to populate said data-containing unit;
• a program to perform said dependent query for each record in said full-backup data file,
• wherein said data-containing unit is populated and pushed into said ETL program,
• wherein referential integrity is maintained for all extracted data that is pushed to said ETL program,
• wherein additional queries are made for all related entity data and said related entity data is pushed into said ETL program,
• wherein each said dependent-query data set is validated and transformed with said ETL program, and
• wherein only each said validated and transformed data set is pushed into said data-warehouse system.

A data-warehouse system wherein there is to be data loaded or restored for use, said system having at least one computing device communicatively coupled with a large dataset, said large dataset having at least one storage device, said at least one storage device containing a full-backup data file and at least one partial-backup data file, said data files containing real-world entity data, said computing device having a Extraction Load Program (ETL) program to subject the data intended to be loaded to a transform stage according to data-warehouse-specific rules, comprising:

• a first partial-backup data file,
• wherein the loading of said first partial-backup data file makes the data in said first partial-backup data file available for use, at a time when no other backup data files have been loaded,
• wherein said first partial-backup data file is one partial-backup data file within a series of historical partial-backup files for said data-warehouse system, and
• wherein said first partial-backup data file is not a full-backup data file, further comprising:
• at least one additional partial-backup data file,
• wherein the loading of each said at least one additional partial-backup data file makes the data in each at least one additional partial-backup data file available for use, at a time when other backup data files are available, but have not been loaded,
• wherein said at least one additional partial-backup data file is any partial-backup data file within a series of historical partial-backup files for said data-warehouse system,
• wherein said at least one additional partial-backup data file is not a full-backup data file, and
• wherein each record to be extracted from said at least one additional partial-backup data file for loading into said data-warehouse system is only allowed to be written to said data-warehouse system if either the record to be written to said data-warehouse system is not already represented in said data-warehouse system or the record to be written to said data-warehouse system is designated as newer than its corresponding existing record,
• wherein the relative age of the record to be written to said data-warehouse system as compared to its corresponding existing record is determined by comparing the relative ordering indexes of the records,
• wherein said relative ordering indexes of the records employs a strategy that breaks a data refresh into a backward-fill migration and a forward-fill migration,
• wherein the relative ordering index value for said backward-fill migration is assigned a value that will always cause a comparison of said value with an actual date of another record to result in said actual date being evaluated as newer, thus allowing the associated record under consideration to be overwritten,
• wherein the first introduction of any record will result in the insertion of said record into said data-warehouse system,
• wherein each new record is assigned a relative ordering index value of the date that said record was pushed into said data-warehouse system,
• wherein during said backward-fill migration, any additional records corresponding to a record already loaded in said data-warehouse system will be ignored,
• wherein during said forward-fill migration, a newer record will always overwrite an existing corresponding record in said data-warehouse system.

A data-warehouse system wherein there is to be data loaded or restored for use, said system having at least one computing device communicatively coupled with a large dataset, said large dataset having at least one storage device, said at least one storage device containing a full-backup data file and at least one partial-backup data file, said data files containing real-world entity data, said computing device having a Extraction Load Program (ETL) program to subject the data intended to be loaded to a transform stage according to data-warehouse-specific rules, comprising:

• a first partial-backup data file,
• wherein the loading of said first partial-backup data file makes the data in said first partial-backup data file available for use, at a time when no other backup data files have been loaded,
• wherein said first partial-backup data file is one partial-backup data file within a series of historical partial-backup files for said data-warehouse system, and
• wherein said first partial-backup data file is not a full-backup data file, further comprising:
• a full-backup data file,
• wherein each record to be extracted from said full-backup data file for loading into said data-warehouse system is only allowed to be written to said data-warehouse system if either the record to be written to said data-warehouse system is not already represented in said data-warehouse system or the record to be written to said data-warehouse system is designated as newer than its corresponding existing record,
• wherein the relative age of the record to be written to said data-warehouse system as compared to its corresponding existing record is determined by comparing the relative ordering indexes of the records,
• wherein said relative ordering indexes of the records employs a strategy that breaks a data refresh into a backward-fill migration and a forward-fill migration,
• wherein the relative ordering index value for said backward-fill migration is assigned a value that will always cause a comparison of said value with an actual date of another record to result in said actual date being evaluated as newer, thus allowing the associated record under consideration to be overwritten,
• wherein the first introduction of any record will result in the insertion of said record into said data-warehouse system,
• wherein each new record is assigned a relative ordering index value of the date that said record was pushed into said data-warehouse system,
• wherein during said backward-fill migration, any additional records corresponding to a record already loaded in said data-warehouse system will be ignored,
• wherein during said forward-fill migration, a newer record will always overwrite an existing corresponding record in said data-warehouse system.

A data-warehouse system wherein there is to be data loaded or restored for use, said system having at least one computing device communicatively coupled with a large dataset, said large dataset having at least one storage device, said at least one storage device containing a full-backup data file and at least one partial-backup data file, said data files containing real-world entity data, said computing device having a Extraction Load Program (ETL) program to subject the data intended to be loaded to a transform stage according to data-warehouse-specific rules, comprising:

• a first partial-backup data file,
• wherein the loading of said first partial-backup data file makes the data in said first partial-backup data file available for use, at a time when no other backup data files have been loaded,
• wherein said first partial-backup data file is one partial-backup data file within a series of historical partial-backup files for said data-warehouse system, and
• wherein said first partial-backup data file is not a full-backup data file, further comprising:
• at least one additional partial-backup data file,
• wherein the loading of each said at least one additional partial-backup data file makes the data in each at least one additional partial-backup data file available for use, at a time when other backup data files are available, but have not been loaded,
• wherein said at least one additional partial-backup data file is any partial-backup data file within a series of historical partial-backup files for said data-warehouse system,
• wherein said at least one additional partial-backup data file is not a full-backup data file, and
• wherein each record to be extracted from said at least one additional partial-backup data file for loading into said data-warehouse system is only allowed to be written to said data-warehouse system if either the record to be written to said data-warehouse system is not already represented in said data-warehouse system or the record to be written to said data-warehouse system is designated as newer than its corresponding existing record, further comprising:
• a full-backup data file,
• wherein each record to be extracted from said full-backup data file for loading into said data-warehouse system is only allowed to be written to said data-warehouse system if either the record to be written to said data-warehouse system is not already represented in said data-warehouse system or the record to be written to said data-warehouse system is designated as newer than its corresponding existing record,
• wherein the relative age of the record to be written to said data-warehouse system as compared to its corresponding existing record is determined by comparing the relative ordering indexes of the records,
• wherein said relative ordering indexes of the records employs a strategy that breaks a data refresh into a backward-fill migration and a forward-fill migration,
• wherein the relative ordering index value for said backward-fill migration is assigned a value that will always cause a comparison of said value with an actual date of another record to result in said actual date being evaluated as newer, thus allowing the associated record under consideration to be overwritten,
• wherein the first introduction of any record will result in the insertion of said record into said data-warehouse system,
• wherein each new record is assigned a relative ordering index value of the date that said record was pushed into said data-warehouse system,
• wherein during said backward-fill migration, any additional records corresponding to a record already loaded in said data-warehouse system will be ignored,
• wherein during said forward-fill migration, a newer record will always overwrite an existing corresponding record in said data-warehouse system.

A data-warehouse system wherein there is to be data loaded or restored for use, said system having at least one computing device communicatively coupled with a large dataset, said large dataset having at least one storage device, said at least one storage device containing a full-backup data file and at least one partial-backup data file, said data files containing real-world entity data, said computing device having a Extraction Load Program (ETL) program to subject the data intended to be loaded to a transform stage according to data-warehouse-specific rules, comprising:

• a first partial-backup data file,
• wherein the loading of said first partial-backup data file makes the data in said first partial-backup data file available for use, at a time when no other backup data files have been loaded,
• wherein said first partial-backup data file is one partial-backup data file within a series of historical partial-backup files for said data-warehouse system, and
• wherein said first partial-backup data file is not a full-backup data file, further comprising:
• at least one additional partial-backup data file,
• wherein the loading of each said at least one additional partial-backup data file makes the data in each at least one additional partial-backup data file available for use, at a time when other backup data files are available, but have not been loaded,
• wherein said at least one additional partial-backup data file is any partial-backup data file within a series of historical partial-backup files for said data-warehouse system,
• wherein said at least one additional partial-backup data file is not a full-backup data file, and
• wherein each record to be extracted from said at least one additional partial-backup data file for loading into said data-warehouse system is only allowed to be written to said data-warehouse system if either the record to be written to said data-warehouse system is not already represented in said data-warehouse system or the record to be written to said data-warehouse system is designated as newer than its corresponding existing record, further comprising:
• a full-backup data file,
• wherein each record to be extracted from said full-backup data file for loading into said data-warehouse system is only allowed to be written to said data-warehouse system if either the record to be written to said data-warehouse system is not already represented in said data-warehouse system or the record to be written to said data-warehouse system is designated as newer than its corresponding existing record,
• wherein the relative age of the record to be written to said data-warehouse system as compared to its corresponding existing record is determined by comparing the relative ordering indexes of the records, wherein a human user selects said first partial-backup data file to be subjected to said backward-fill migration from other available backup data files, when any candidate first partial-backup files do not contain a date field that indicates the relative newness of the data as compared to other available backup data files, by evaluating the physical characteristics and known circumstances surrounding the creation of the backup data file under consideration to be said first partial-backup data file in order to assume a relative age of said selected first partial-backup data file for the purpose of implementing a strategy for employing relative ordering indexes for record loading during said backward-fill migration.