Incremental Clustering Of Database Tables
AssigneeSnowflake Inc.
InventorsJiaqi Yan, Thierry Cruanes, Jeffrey Rosen, William Waddington, Prasanna Rajaperumal, Abdul Munir
Life of this patent
Estimating the term…
Abstract
Automatic clustering of a database table is disclosed. A method for automatic clustering of a database table includes receiving an indication that a data modification task has been executed on a table and determining whether the table is sufficiently clustered. The method includes, in response to determining the table is not sufficiently clustered, selecting one or more micro-partitions of the table to be reclustered. The method includes assigning each of the one or more micro-partitions to an execution node to be reclustered.
Claims25
A method comprising:
- receiving an indication that a data modification task has been executed on a table;
- determining whether the table is sufficiently clustered;
- in response to determining the table is not sufficiently clustered, selecting one or more micro-partitions of the table to be reclustered;
- and
- assigning each of the one or more micro-partitions to an execution node to be reclustered.
A system comprising:
- a compute service manager for managing internal operations of a cloud-based database platform;
- a plurality of shared storage devices collectively storing database data, wherein the plurality of shared storage devices is independent of the compute service manager;
- and
- an execution platform comprising a plurality of execution nodes, wherein the execution platform is independent of the plurality of shared storage devices and the compute service manager;
- wherein the compute service manager is configured to:
- receive an indication that a data modification task has been executed on a table of the database by one or more execution nodes of the execution platform;
- determine whether the table is sufficiently clustered;
- in response to determining the table is not sufficiently clustered, select one or more micro-partitions of the table to be reclustered;
- and
- assign each of the one or more micro-partitions to an execution node to be reclustered.
The system of 11, wherein the compute service manager is further configured to define a budget for allocating processing resources to performing reclustering operations, and wherein the determining whether the table is sufficiently clustered is based at least in part on the budget.
The system of 11, wherein the compute service manager is further configured to partition the one or more micro-partitions of the table to be reclustered into one or more batches each comprising a grouping of micro-partitions to be reclustered.
The system of 11, wherein the compute service manager is configured to select the one or more micro-partitions of the table to be reclustered by:
- determining a maximum number of levels for the table based at least on a size of the table;
- dividing the table into levels;
- selecting a macro-batch of micro-partitions within each level, wherein the macro-batch centers around a single peak and comprises defined boundaries;
- and
- selecting micro-partitions from the macro-batch.
The system of 11, wherein the data modification task comprises ingesting new micro-partitions into the table and wherein the compute service manager is configured to determine whether the table is sufficiently clustered by:
- retrieving level information for the table;
- identifying a proportion of micro-partitions in lower levels of the table;
- determining whether a high proportion of micro-partitions are in the lower levels;
- in response to determining a high proportion of micro-partitions are not in the lower levels, entering a stable mode in which no reclustering operations will not be performed;
- and
- in response to determining a high proportion of micro-partitions are in the lower levels, entering a catch-up mode in which reclustering operations will be performed.
A processor configurable to execute instructions stored in non-transitory computer readable storage media, the instructions comprising:
- receiving an indication that a data modification task has been executed on a table;
- determining whether the table is sufficiently clustered;
- in response to determining the table is not sufficiently clustered, selecting one or more micro-partitions of the table to be reclustered;
- and
- assigning each of the one or more micro-partitions to an execution node to be reclustered.
Dependencies are inferred from the claim text.