US 8,479,158 B2Granted

Business information warehouse toolkit and language for warehousing simplification and automation

AssigneeInternational Business Machines Corporation

InventorsYing Chen, Bin He, Ana Lelescu, James J. Rhodes, Rui Wang

Life of this patent

Estimating the term…

Loading patent…

Abstract

A method for use with an information (or data) warehouse comprises managing the information warehouse with instructions in a declarative language. The instructions specify information warehouse-level tasks to be done without specifying certain details of how the tasks are to be implemented, for example, using databases and text indexers. The details are hidden from the user and include, for example, in an information warehouse having a FACT table that joins two or more dimension tables, details of database level operations when structured data are being handled, including database command line utilities, database drivers, and structured query language (SQL) statements; and details of text-indexing engines when unstructured data are being handled. The information warehouse is managed in a dynamic way in which different tasks—such as data loading tasks and information warehouse construction tasks—may be interleaved (i.e., there is no particular order in which the different tasks must be completed).

Claims26

Dependent claims

A method for information warehouse construction in memory of a hardware computer processor, comprising:

accepting input from a user at a front-end graphical user interface;
generating a plurality of commands employing declarative statements in an information warehouse language and performing loading or copying of data using:
information warehouse (IW) level commands that when implemented provide support for both structured and unstructured data,
indexing engine level commands configured to perform tasks specified by the commands from a user interface,
source mapping commands configured to perform at least one of:
a data format transformation, source-to-target complex schema matching, and complex data value mapping,
data loading commands configured to perform at least one of:
a data format transformation, source-to-target complex schema matching, and complex data value mapping, and
dimension level commands that when implemented provide support for both structured and unstructured data and include at least one of:
creating a dimension, altering the dimension, and dropping the dimension;
and
constructing an information warehouse based on a script comprising the IW level commands and the dimension level commands according to the language specification for information warehouse construction;
and
providing error correction for data loaded from a source file into the information warehouse using:
an abort command that uses a pending log to return the information warehouse to a consistent break point prior to a data load into the information warehouse so that operational details of maintaining data integrity for both the structured and the unstructured data are hidden from a user of the abort command, such that a load command automatically resumes loading data into the information warehouse from a last consistent breakpoint of a data load, while maintaining data integrity for both the structured and the unstructured data by performing system level tasks that are hidden from a user of the load command;
an undo command, operating with regard to the source file, that uses a checkpoint log to automatically return the information warehouse to a state of the information warehouse prior to loading of the source file, while maintaining data integrity for both the structured and the unstructured data by performing system level tasks that are hidden from a user of the undo command;
and
a redo command, operating with regard to the source file, that uses the checkpoint log to automatically return the information warehouse to a state of the information warehouse after loading of the source file, wherein the loading of the source file has previously been undone, and while maintaining data integrity for both the structured and the unstructured data by performing system level tasks that are hidden from a user of the redo command.

15 more elements

A method for data loading in an information warehouse stored in memory of a hardware computer processor ,comprising:

accepting input from a user at a front-end graphical user interface;
generating a plurality of commands employing declarative statements in an information warehouse language and performing loading or copying of the data using:
information warehouse (IW) level commands that when implemented provide support for both structured and unstructured data,
indexing engine level commands,
dimension level commands that when implemented include at least one of:
creating a dimension, altering a dimension, and dropping the dimension, and provide support for both structured and unstructured data,
source mapping commands configured to perform at least one of:
performing a data format transformation, performing source-to-target complex schema matching, and performing complex data value matching, and
data loading commands configured to perform at least one of:
performing a data format transformation, performing source-to-target complex schema matching, and performing complex data value matching;
loading data into an information warehouse based on a script including both the source mapping commands and the data loading commands according to the language specification for data loading;
and
providing error correction for data loaded from a source file into the information warehouse according to at least one of:
the information warehouse level commands, the dimension level commands, the source mapping commands and the data loading commands,
an abort command that uses a pending log to return the information warehouse to a consistent break point prior to a data load into the information warehouse so that operational details of maintaining data integrity for both the structured and the unstructured data are hidden from a user of the abort command, such that a load command automatically resumes loading data into the information warehouse from a last consistent breakpoint of a data load, while maintaining data integrity for both the structured and the unstructured data by performing system level tasks that are hidden from a user of the load command;
an undo command, operating with regard to the source file, that uses a checkpoint log to automatically return the information warehouse to a state of the information warehouse prior to loading of the source file, while maintaining data integrity for both the structured and the unstructured data by performing system level tasks that are hidden from a user of the undo command;
and
a redo command, operating with regard to the source file, that uses the checkpoint log to automatically return the information warehouse to a state of the information warehouse after loading of the source file, wherein the loading of the source file has previously been undone, and while maintaining data integrity for both the structured and the unstructured data by performing system level tasks that are hidden from a user of the redo command.

15 more elements

A method for data maintenance of an information warehouse stored in memory of a hardware computer processor, comprising:

providing a language specification for information warehouse (IW) maintenance in the information warehouse;
accepting input from a user at a front-end graphical user interface;
generating a plurality of commands employing declarative statements in an information warehouse language and performing loading or copying by:
implementing indexing engine level commands configured to perform tasks specified by the commands from an interface;
implementing dimension level commands that when implemented provide support for both structured and unstructured data and include at least one of:
creating a dimension, altering the dimension, and dropping the dimension;
implementing IW level commands that when implemented provide support for both structured and unstructured data;
implementing source mapping commands configured to perform at least one of:
a data transformation, source-to-target complex schema matching, and complex data value mapping;
and
implementing data loading commands configured to perform at least one of:
a data format transformation, source-to-target complex schema matching, and complex data value mapping;
and
performing an IW maintenance operation on the information warehouse based on a script including at least one of failure recovery commands, error correction commands according to the language specification for IW maintenance,
an abort command that uses a pending log to return the information warehouse to a consistent break point prior to a data load into the information warehouse so that operational details of maintaining data integrity for both structured and unstructured data are hidden from a user of the abort command, such that a load command automatically resumes loading data into the information warehouse from a last consistent breakpoint of a data load, while maintaining data integrity for both the structured and the unstructured data by performing system level tasks that are hidden from a user of the load command,
an undo command, operating with regard to a source file, that uses a checkpoint log to automatically return the information warehouse to a state of the information warehouse prior to loading of the source file, while maintaining data integrity for both the structured and the unstructured data by performing system level tasks that are hidden from a user of the undo command, and
a redo command, operating with regard to the source file, that uses the checkpoint log to automatically return the information warehouse to a state of the information warehouse after loading of the source file, wherein the loading of the source file has previously been undone, and while maintaining data integrity for both the structured and the unstructured data by performing system level tasks that are hidden from a user of the redo command.

14 more elements

An information warehouse system comprising:

a hardware computer processor including memory;
a front-end graphical user interface that accepts inputs from a user and generates a plurality of commands employing declarative statements in an information warehouse language;
an information warehouse language processor stored on the memory that receives a plurality of user interface commands from the user interface and performs loading using at least one of:
system level commands;
database level commands that when implemented provide support for both structured and unstructured data;
indexing engine level commands configured to perform tasks specified by the commands from the interface;
source mapping commands configured to perform at least one of:
a data transformation, source-to-target complex schema matching, and complex data value mapping;
data loading commands configured to perform at least one of:
a data format transformation, source-to-target complex schema matching, and complex data value mapping;
and
dimension level commands that when implemented provide support for both structured and unstructured data and include at least one of:
creating a dimension, altering the dimension, and dropping the dimension;
a warehouse builder that:
receives the at least one of:
the system level commands, the database level commands, and the indexing engine level commands;
performs tasks specified by the at least one of system level, database level, and indexing engine level commands;
builds an information warehouse;
and
provides error correction for data loaded from a source file into the information warehouse using:
an abort command that uses a pending log to return the information warehouse to a consistent break point prior to a data load into the information warehouse so that operational details of maintaining data integrity for both the structured and the unstructured data are hidden from a user of the abort command, such that a load command automatically resumes loading data into the information warehouse from a last consistent breakpoint of a data load, while maintaining data integrity for both the structured and the unstructured data by performing system level tasks that are hidden from a user of the load command;
an undo command, operating with regard to the source file, that uses a checkpoint log to automatically return the information warehouse to a state of the information warehouse prior to loading of the source file, while maintaining data integrity for both structured and unstructured data by performing system level tasks that are hidden from a user of the undo command;
and
a redo command, operating with regard to the source file, that uses the checkpoint log to automatically return the information warehouse to a state of the information warehouse after loading of the source file, wherein the loading of the source file has previously been undone, and while maintaining data integrity for both the structured and the unstructured data by performing system level tasks that are hidden from a user of the redo command.

21 more elements

A computer program product for use with an information warehouse, the computer program product comprising a non-transitory computer readable medium including a computer executable program, wherein the computer executable program when executed on a computer causes the computer to:

accept commands of an information warehouse language using declarative statements provided by a script;
parse the commands from the script;
generate data description language commands from the parsed commands including information warehouse level commands that when implemented provide support for both structured and unstructured data;
execute the data description language commands to construct an information warehouse;
perform loading or copying of the structured and unstructured data, using:
indexing engine level commands configured to perform tasks specified by commands from an interface;
source mapping commands configured to perform at least one of:
a data format transformation, source-to-target complex schema matching, and complex data value mapping;
data loading commands configured to perform at least one of:
a data format transformation, source-to-target complex schema matching, and complex data value mapping;
dimension level commands that when implemented include at least one of:
creating a dimension, altering the dimension, and dropping the dimension, and provide support for both structured and unstructured data;
and
provide error correction for data loaded from a source file into the information warehouse using:
an abort command that uses a pending log to return the information warehouse to a consistent break point prior to a data load into the information warehouse so that operational details of maintaining data integrity for both the structured and the unstructured data are hidden from a user of the abort command, such that a load command automatically resumes loading data into the information warehouse from a last consistent breakpoint of a data load, while maintaining data integrity for both the structured and the unstructured data by performing system level tasks that are hidden from a user of the load command;
an undo command, operating with regard to a source file, that uses a checkpoint log to automatically return the information warehouse to a state of the information warehouse prior to loading of the source file, while maintaining data integrity for both structured and unstructured data by performing system level tasks that are hidden from a user of the undo command;
and
a redo command, operating with regard to the source file, that uses the checkpoint log to automatically return the information warehouse to a state of the information warehouse after loading of the source file, wherein the loading of the source file has previously been undone, and while maintaining data integrity for both structured and unstructured data by performing system level tasks that are hidden from a user of the redo command.

15 more elements

A computer program product for use with an information warehouse, the computer program product comprising a non-transitory computer readable medium including a computer executable program, wherein the computer readable program when executed on a computer causes the computer to:

accept commands of an information warehouse language using declarative statements provided by a script;
parse the commands from the script;
extract desired data from a source file according to the parsed commands;
transform the extracted data into a target format according to the parsed commands;
load the transformed data into information warehouse tables, file system locations according to the parsed commands;
perform loading or copying of the transformed data, using:
information warehouse level commands that provide support for both structured and unstructured data,
indexing engine level commands configured to perform tasks specified by the commands from an interface;
source mapping commands configured to perform at least one of:
a data format transformation, source-to-target complex schema matching, and complex data value mapping;
data loading commands configured to perform at least one of:
a data format transformation, source-to-target complex schema matching, and complex data value mapping;
and
dimension level commands that include at least one of:
creating a dimension, altering the dimension, and dropping the dimension and provide support for both structured and unstructured data;
and
provide error correction for data loaded from a source file into the information warehouse using:
an abort command that uses a pending log to return the information warehouse to a consistent break point prior to a data load into the information warehouse so that operational details of maintaining data integrity for both the structured and the unstructured data are hidden from a user of the abort command, such that a load command automatically resumes loading data into the information warehouse from a last consistent breakpoint of a data load, while maintaining data integrity for both the structured and the unstructured data by performing system level tasks that are hidden from a user of the load command;
an undo command, operating with regard to the source file, that uses a checkpoint log to automatically return the information warehouse to a state of the information warehouse prior to loading of the source file, while maintaining data integrity for both the structured and the unstructured data by performing system level tasks that are hidden from a user of the undo command;
and
a redo command, operating with regard to the source file, that uses the checkpoint log to automatically return the information warehouse to a state of the information warehouse after loading of the source file, wherein the loading of the source file has previously been undone, and while maintaining data integrity for both the structured and the unstructured data by performing system level tasks that are hidden from a user of the redo command.

18 more elements

Dependencies are inferred from the claim text.

Facts

Application: 12/134,011Filed 5 Jun 2008 · continuation
Priority: 7 Jun 2007
Published: US 2008/0306987 A111 Dec 2008
Granted: 2 Jul 2013
Term ends: Not estimatedThe source records don’t support an estimate.

Classification6 codes

G06F 11/1471
Error detection or correction of the data by redundancy in operations: involving logging of persistent data for recovery
G06F 16/283
Databases characterised by their database models, e.g. relational or object models: multi-dimensional databases or data warehouses, e.g. MOLAP or…
G06Q 30/02
Commerce: marketing; Price estimation or determination; Fundraising
G06F 2201/80
Indexing scheme relating to error detection, to error correction, and to monitoring: database-specific techniques

Family

Application11/759,851Parent of this continuation · filed 7 Jun 2007
US 2008/0306987 A1Pre-grant publication · published 11 Dec 2008
US 2008/0307386 A1Published application
US 8,056,054 B2Related patent
US 8,479,158 B2This patent

From the US filing history. Foreign family members aren’t listed.