Hadoop OLAP engine
AssigneeeBay Inc.
InventorsLuke Qing Han, Xu Jiang, Song Yi, Chauncey Li
Life of this patent
Estimating the term…
Abstract
In various example embodiments, systems and methods for building data cubes to be stored in a cube store are presented. In some embodiments, a metadata engine generates the cube metadata. In further embodiments, cube data is generated by a cube build engine based on the cube metadata and source data. The cube build engine performs a multi-stage MapReduce job on the source data to produce a multi-dimensional cube lattice having multiple cuboids. In further embodiments, the cube data is provided to the cube store.
Claims20
A method implemented by a Hadoop Online Analytical Processing (OLAP) engine, the method comprising:
- accessing a query history of a user to identify a query pattern;
- automatically identifying cube metadata based on the query pattern identified from the query history, wherein the cube metadata defines dimensions and measure information for a cube;
- receiving the cube metadata;
- building the cube based on the cube metadata and source data;
- storing the cube in one or more HBase tables of a cube store, the cube store configured for access by a query engine, and at least one HBase table being distributed over a plurality of servers;
- partitioning the cube stored in the cube store into a first data partition configured as a cube and a second data partition configured as a cube, the first data partition configured as a cube corresponding to a first time slot and the second data partition configured as a cube corresponding to a second time slot that represents a smaller amount of time than the first time slot;
- and
- refreshing the first data partition configured as a cube by periodically merging cube data in the second data partition configured as a cube with cube data in the first data partition configured as a cube.
A machine readable medium not having any transitory signals and storing instructions that, when executed by at least one processor of a machine, cause the machine implement a Hadoop Online Analytical Processing (OLAP) engine to perform operations comprising:
- accessing a query history of a user to identify a query pattern;
- automatically identifying cube metadata based on the query pattern identified from the query history, wherein the cube metadata defines dimensions and measure information for a cube;
- receiving the cube metadata;
- building the cube based on the cube metadata and source data;
- storing the cube in one or more HBase tables of a cube store, the cube store configured for access by a query engine, and at least one HBase table being distributed over a plurality of servers;
- partitioning the cube stored in the cube store into a first data partition configured as a cube and a second data partition configured as a cube, the first data partition configured as a cube corresponding to a first time slot and the second data partition configured as a cube corresponding to a second time slot that represents a smaller amount of time than the first time slot;
- and
- refreshing the first data partition configured as a cube by periodically merging cube data in the second data partition configured as a cube with cube data in the first data partition configured as a cube.
A computer system, comprising:
- at least one processor;
- and
- a machine readable medium not having any transitory signals and storing instructions that, when executed by the at least one processor, cause the computer system to implement a Hadoop Online Analytical Processing (OLAP) engine perform operations comprising:
- accessing a query history of a user to identify a query pattern;
- automatically identifying cube metadata based on the query pattern identified from the query history, wherein the cube metadata defines dimensions and measure information for a cube;
- receiving the cube metadata;
- building the cube based on the cube metadata and source data;
- storing the cube in one or more HBase tables of a cube store, the cube store configured for access by a query engine, and at least one HBase table being distributed over a plurality of servers;
- partitioning the cube stored in the cube store into a first data partition configured as a cube and a second data partition configured as a cube, the first data partition configured as a cube corresponding to a first time slot and the second data partition configured as a cube corresponding to a second time slot that represents a smaller amount of time than the first time slot;
- and
- refreshing the first data partition configured as a cube by periodically merging cube data in the second data partition configured as a cube with cube data in the first data partition configured as a cube.
Dependencies are inferred from the claim text.