Meta-data driven data ingestion using MapReduce framework
AssigneeTurn Inc.
InventorsMingxi Wu, Songting Chen
Life of this patent
Estimating the term…
Abstract
A generic approach for automatically ingesting data into an HDFS (Hadoop File System) based data warehouse includes a datahub server, a generic pipelined data loading framework, and a meta-data model that, together, address data loading efficiency, data source heterogeneities, and data warehouse schema evolvement. The loading efficiency is achieved via the MapReduce scale-out solution. The meta-data model is comprised of configuration files and a catalog. The configuration file is setup per ingestion task. The catalog manages the data warehouse schema. When a scheduled data loading task is executed, the configuration files and the catalog collaboratively drive the datahub server to load the heterogeneous data to their destination schemas automatically.
Claims27
A method for automatically ingesting data into a data warehouse, comprising:
- providing a datahub server for executing data loading tasks;
- providing a generic pipelined data loading framework that leverages a MapReduce environment for ingestion of a plurality of heterogeneous data sources;
- and
- providing a processor implemented meta-data model comprised of a plurality of configuration files and a catalog;
- wherein a configuration file is setup per ingestion task;
- wherein said catalog manages data warehouse schema;
- wherein when a scheduled data loading task is executed by said datahub server;
- and
- wherein said configuration files and said catalog collaboratively drive the datahub server to load the heterogeneous data to their destination schemas automatically and independently of data source heterogeneities and data warehouse schema evolvement.
An apparatus for automatically ingesting data into a data warehouse, comprising:
- a datahub server for executing data loading tasks comprising heterogeneous data received from a plurality of different servers;
- a generic pipelined data loading framework that leverages a MapReduce environment for ingestion of a plurality of heterogeneous data sources;
- and
- a processor implemented meta-data model comprised of a plurality of configuration files and a catalog;
- wherein a configuration file is setup per ingestion task;
- wherein said catalog manages data warehouse schema;
- wherein when a scheduled data loading task is executed by said datahub server;
- and
- wherein said configuration files and said catalog collaboratively drive the datahub server to load the heterogeneous data to their destination schemas automatically and independently of data source heterogeneities and data warehouse schema evolvement into a Hadoop cluster.
A method for automatically ingesting data into a data warehouse, comprising:
- providing a datahub server for executing data loading tasks;
- providing a generic pipelined data loading framework leverages a MapReduce environment for ingestion of a plurality of heterogeneous data sources;
- and
- providing a processor implemented meta-data model comprised of a plurality of configuration files and a catalog;
- wherein a configuration file is setup per ingestion task;
- wherein said catalog manages data warehouse schema;
- wherein when a scheduled data loading task is executed by said datahub server;
- wherein said configuration files and said catalog collaboratively drive the datahub server to load the heterogeneous data to their destination schemas automatically and independently of data source heterogeneities and data warehouse schema evolvement;
- said datahub server executing said data loading task by:
- downloading and transforming a job running on said datahub server by referring to a configuration file and pipeline status files to determine where, what, and how to download the data source files to a local working directory, and then transforming the files.
A datahub server, comprising:
- a processor implemented framework for leveraging a MapReduce environment to route source data to a destination;
- said framework consulting meta-data to carry out different instances of a pipeline to perform ingestion tasks;
- wherein meta-data modeling during data ingestion comprises destination schema modeling via a catalog, and client configuration modeling per ingestion task via a configuration file.
A process, comprising:
- providing a processor implemented framework for leveraging a MapReduce environment to route source data to a destination;
- said framework consulting meta-data to carry out different instances of a pipeline to perform ingestion tasks;
- wherein meta-data modeling during data ingestion comprises destination schema modeling via a catalog, and client configuration modeling per ingestion task via a configuration file.
A method for automatically ingesting data into a data warehouse, comprising:
- providing a datahub server for following one or more configuration files to load a plurality of heterogeneous sources of data into a Hadoop file system (HDFS);
- said datahub server launching a MapReduce job to join all of said heterogeneous data sources with existing data in a common destination schema;
- and
- said datahub server performing a join task to join client data with existing data of a same schema by launching a MapReduce job which reads all newly arrived data and the existing data of the destination schema, and which performs the join in a reducer of a MapReduce framework.
An apparatus for automatically ingesting data into a data warehouse, comprising:
- a datahub server following one or more configuration files to load a plurality of heterogeneous sources of data into a Hadoop file system (HDFS);
- said datahub server launching a MapReduce job to join all of said heterogeneous data sources with existing data in a common destination schema;
- and
- said datahub server performing a join task to join client data with existing data of a same schema by launching a MapReduce job which reads all newly arrived data and the existing data of the destination schema, and which performs the join in a reducer of a MapReduce framework.
Dependencies are inferred from the claim text.