Back to results
US 11,455,308 B2Granted
Partition aware partial query processing
AssigneeSap SE
InventorsJulian Schwing, Johannes Merx, Christoph Weyerhaeuser
Life of this patent
Estimating the term…
Abstract
Described herein includes a calculation scenario of a calculation engine that efficiently partitions data for processing at separate hosts, including in parallel, and unions intermediate results from such separate processing when required for further processing. Such parallel processing of partitions can allow for faster processing times, and such unioning of data only when required for further processing can limit the transferring of data that results in slower processing.
Claims18
A system comprising:
- at least one processor;
- and
- at least one memory including instructions which when executed by the at least one processor cause operations comprising:
- detecting a partitioning flag at a first datasource operation in a calculation scenario to be executed in a calculation engine of a database management system, the partitioning flag being set at a design time of the calculation scenario and providing instructions to the calculation scenario regarding parallel processing of table partitions at a runtime of the calculation scenario, the calculation scenario defining a common structure for execution of a plurality of queries having one or more first attributes, wherein the calculation scenario defines one or more parameterized calculation schemas for the defined common structure for execution of the plurality of queries, the one or more parameterized calculations schemas including one or more aggregation nodes identifying one or more dimensions for aggregating of one or more table partitions;
- determining, based upon the detected partitioning flag, that a table in a plurality of tables called by the datasource operator is partitioned into a plurality of partitions on a plurality of computing systems each comprising a programmable processor, each table in the plurality of tables being assigned a role and being joined in accordance with a predetermined schema using one or more metamodels in a plurality of metamodels;
- duplicating at least one operator, including a table datasource operator, per partition in the plurality of partitions, in the calculation scenario that follows the partitioning flag to form a plurality of copies of the at least one operator such that one copy of the at least one operator is provided to each computing system of the plurality of computing systems for execution on a respective partition of the plurality of partitions to generate an intermediate result for each partition;
- parallel processing, by each respective computing system, each partition in the plurality of partitions, from each partition's table data source operator to a second flag indicating a union of the intermediate results, and in accordance with a physical location of each partition, without changing the calculation scenario and generating the intermediate result for each partition;
- detecting, at the runtime of the calculation scenario, the second flag indicating the union of the intermediate results;
- unioning the intermediate results, the unioning including:
- defining one or more specific dimensions in the one or more dimensions for aggregating and filtering of the intermediate results;
- selecting, based on the defined one or more specific dimensions, a computing system in the plurality of computing systems;
- transferring the intermediate results to the selected computing system;
- and
- unioning the transferred intermediate results by the selected computing system in accordance with the defined one or more specific dimensions;
- and
- executing, by reusing the defined common structure of the calculation scenario and using the unioned intermediate results, a query in the plurality of queries having one or more second attributes being different from the one or more first attributes, wherein execution of the query using the calculation scenario includes transforming one or more nodes of the calculation scenario not used by the calculation scenario to execute the query into one or more logical database execution plans and combining the one or more logical database execution plans into a single database execution plan for execution of the query.
13 more elements
A computer program product comprising a non-transitory computer-readable storage medium storing instructions that, when executed by at least one programmable processor, cause the at least one programmable processor to perform operations comprising:
- detecting a partitioning flag at a first datasource operation in a calculation scenario to be executed in a calculation engine of a database management system, the partitioning flag being set at a design time of the calculation scenario and providing instructions to the calculation scenario regarding parallel processing of table partitions at a runtime of the calculation scenario, the calculation scenario defining a common structure for execution of a plurality of queries having one or more first attributes, wherein the calculation scenario defines one or more parameterized calculation schemas for the defined common structure for execution of the plurality of queries, the one or more parameterized calculations schemas including one or more aggregation nodes identifying one or more dimensions for aggregating of one or more table partitions;
- determining, based upon the detected partitioning flag, that a table in a plurality of tables called by the datasource operator is partitioned into a plurality of partitions on a plurality of computing systems each comprising a programmable processor, each table in the plurality of tables being assigned a role and being joined in accordance with a predetermined schema using one or more metamodels in a plurality of metamodels;
- duplicating at least one operator, including a table datasource operator, per partition in the plurality of partitions, in the calculation scenario that follows the partitioning flag to form a plurality of copies of the at least one operator such that one copy of the at least one operator is provided to each computing system of the plurality of computing systems for execution on a respective partition of the plurality of partitions to generate an intermediate result for each partition;
- parallel processing, by each respective computing system, each partition in the plurality of partitions, from each partition's table data source operator to a second flag indicating a union of the intermediate results, and in accordance with a physical location of each partition, without changing the calculation scenario and generating the intermediate result for each partition;
- detecting, at the runtime of the calculation scenario, the second flag indicating the union of the intermediate results;
- unioning the intermediate results, the unioning including:
- defining one or more specific dimensions in the one or more dimensions for aggregating and filtering of the intermediate results;
- selecting, based on the defined one or more specific dimensions, a computing system in the plurality of computing systems;
- transferring the intermediate results to the selected computing system;
- and
- unioning the transferred intermediate results by the selected computing system in accordance with the defined one or more specific dimensions;
- and
- executing, by reusing the defined common structure of the calculation scenario and using the unioned intermediate results, a query in the plurality of queries having one or more second attributes being different from the one or more first attributes, wherein execution of the query using the calculation scenario includes transforming one or more nodes of the calculation scenario not used by the calculation scenario to execute the query into one or more logical database execution plans and combining the one or more logical database execution plans into a single database execution plan for execution of the query.
10 more elements
A method for implementation by one or more data processors forming part of at least one computing system, the method comprising:
- detecting a partitioning flag at a first datasource operation in a calculation scenario to be executed in a calculation engine of a database management system, the partitioning flag being set at a design time of the calculation scenario and providing instructions to the calculation scenario regarding parallel processing of table partitions at a runtime of the calculation scenario, the calculation scenario defining a common structure for execution of a plurality of queries having one or more first attributes, wherein the calculation scenario defines one or more parameterized calculation schemas for the defined common structure for execution of the plurality of queries, the one or more parameterized calculations schemas including one or more aggregation nodes identifying one or more dimensions for aggregating of one or more table partitions;
- determining, based upon the detected partitioning flag, that a table in a plurality of tables called by the datasource operator is partitioned into a plurality of partitions on a plurality of computing systems each comprising a programmable processor, each table in the plurality of tables being assigned a role and being joined in accordance with a predetermined schema using one or more metamodels in a plurality of metamodels;
- duplicating at least one operator, including a table datasource operator, per partition in the plurality of partitions, in the calculation scenario that follows the partitioning flag to form a plurality of copies of the at least one operator such that one copy of the at least one operator is provided to each computing system of the plurality of computing systems for execution on a respective partition of the plurality of partitions to generate an intermediate result for each partition;
- parallel processing, by each respective computing system, each partition in the plurality of partitions, from each partition's table data source operator to a second flag indicating a union of the intermediate results, and in accordance with a physical location of each partition, without changing the calculation scenario and generating the intermediate result for each partition;
- detecting, at the runtime of the calculation scenario, the second flag indicating the union of the intermediate results;
- unioning the intermediate results, the unioning including:
- defining one or more specific dimensions in the one or more dimensions for aggregating and filtering of the intermediate results;
- selecting, based on the defined one or more specific dimensions, a computing system in the plurality of computing systems;
- transferring the intermediate results to the selected computing system;
- and
- unioning the transferred intermediate results by the selected computing system in accordance with the defined one or more specific dimensions;
- and
- executing, by reusing the defined common structure of the calculation scenario and using the unioned intermediate results, a query in the plurality of queries having one or more second attributes being different from the one or more first attributes, wherein execution of the query using the calculation scenario includes transforming one or more nodes of the calculation scenario not used by the calculation scenario to execute the query into one or more logical database execution plans and combining the one or more logical database execution plans into a single database execution plan for execution of the query.
10 more elements
Dependencies are inferred from the claim text.