System and method for generating rowid range-based splits in a massively parallel or distributed database environment

A system for database split generation in a massively parallel or distributed database environment including a plurality of databases and a data warehouse layer providing data summarization and querying of the plurality of databases, the system comprising:

• one or more microprocessors;
• and
• a database table accessor running on the one or more microprocessors, wherein the database table accessor operates to perform steps comprising:
• obtaining, from an associated client application, a client application original query for blocks of data in a table of the data warehouse layer, the client application original query comprising one or more of query data and/or user preference data, the query data being representative of a user query that selects all columns in the table of the data warehouse layer, and the user preference data being representative of a user preference for a ROWID-based splits generator;
• obtaining, from the data warehouse layer in accordance with the client application original query for blocks of data in the table, table data representative of one or more properties of the table of the data warehouse layer, the table data comprising files, data blocks, and row numbers for the table;
• creating multiple block ranges for the blocks of data in the table in accordance with the table data, wherein each block range has a fixed size of blocks;
• creating ROWID range data for each of the multiple block ranges, the ROWID range data being representative of pseudo columns of the table that serve as unique identifiers for a row of the table;
• selecting a ROWID-based splits generator based on the data in the table being of a known database chunk size and based on the client application original query comprising the one or more of the query data and/or the user preference data;
• using the selected ROWID-based splits generator, generating table splits, the table splits dividing the client application original query into a plurality of ROWID query splits wherein each ROWID query split is associated with a one of the multiple block ranges in accordance with the ROWID range data of the respective one of the multiple block ranges;
• and
• outputting the plurality of ROWID query splits to a plurality of associated mappers for execution by each of the plurality of associated mappers of the plurality of ROWID query splits against a corresponding block range of the data in the table of the date warehouse layer.

A non-transitory computer readable storage medium, including instructions stored thereon which when read and executed by one or more computers of a database table accessor in a massively parallel or distributed database environment including a plurality of databases and a data warehouse layer providing data summarization and querying of the plurality of databases, cause the one or more computers of the database table accessor to perform the steps comprising:

• obtaining, from an associated client application, a client application original query for blocks of data in a table of the data warehouse layer, the client application original query comprising one or more of query data and/or user preference data, the query data being representative of a user query that selects all columns in the table of the data warehouse layer, and the user preference data being representative of a user preference for a ROWID-based splits generator;
• obtaining, from the data warehouse layer in accordance with the client application original query for blocks of data in the table, table data representative of one or more properties of the table of the data warehouse layer, the table data comprising files, data blocks, and row numbers for the table;
• creating multiple block ranges for the blocks of data in the table in accordance with the table data, wherein each block range has a fixed size of blocks;
• creating ROWID range data for each of the multiple block ranges, the ROWID range data being representative of pseudo columns of the table that serve as unique identifiers for a row of the table;
• selecting a ROWID-based splits generator based on the data in the table being of a known database chunk size and based on the client application original query comprising the one or more of the query data and/or the user preference data;
• using the selected ROWID-based splits generator, generating table splits, the table splits dividing the client application original query into a plurality of ROWID query splits wherein each ROWID query split is associated with a one of the multiple block ranges in accordance with the ROWID range data of the respective one of the multiple block ranges;
• and
• outputting the plurality of ROWID query splits to a plurality of associated mappers for execution by each of the plurality of associated mappers of a ROWID query split against a corresponding block range of the data in the table of the data warehouse layer.