US 11,537,613 B1Granted

Merge small file consolidation

AssigneeSnowflake Inc.

InventorsThierry Cruanes, Varun Ganesh, Ryan Michael Thomas Shelly, Jiaqi Yan

Life of this patent

Estimating the term…

Loading patent…

Abstract

The subject technology receives a query plan corresponding to a query. The subject technology executes the query based at least in part on the query plan, the executing including: filtering a first set of files that are to be modified by a merge statement, performing a split operation to send information related to a second set of files to a scan set builder operation in a first portion of the query plan and scan back operation in a second portion of the query plan, performing the scan set builder operation to remove the second set of files from the first set of files, performing a table scan operation based on a third set of files, and performing a first union all operation to combine the first set of data with a second set of data as a first set of combined data.

Claims30

Dependent claims

A network-based database system comprising:

at least one hardware processor;
and
a memory storing instructions that cause the at least one hardware processor to perform operations comprising:
receiving a query plan corresponding to a query, the query plan comprising a tree structure including a set of nodes, each node corresponding to an operation to be performed when executing the query;
executing the query based at least in part on the query plan, the executing comprising:
filtering a first set of files that are to be modified by a merge statement, the filtering generating a second set of files comprising files from the first set of files that are to be modified by the merge statement;
performing a split operation to send information related to the second set of files to a scan set builder operation in a first portion of the query plan and scan back operation in a second portion of the query plan, the first portion comprising a first branch of the tree structure, the second portion comprising a second branch of the tree structure, each branch including a different set of nodes corresponding to different operations performed when executing the query;
performing the scan set builder operation in the first branch of the tree structure to remove the second set of files from the first set of files, the scan set builder operation generating a third set of files, the third set of files comprising candidate files for consolidation and defragmentation from a source table that are not to be unregistered by the merge statement;
performing, using at least a table scan operation, the consolidation and defragmentation of the third set of files, the table scan operation generating a first set of data from scanning the third set of files, the consolidation and defragmentation of the third set of files reducing fragmentation of the source table and reducing an overhead with managing and storing metadata associated with the third set of files, and the scan set builder operation corresponding to a first node below a second node corresponding to the table scan operation in the tree structure corresponding to the query plan;
and
performing a first union all operation to combine the first set of data with a second set of data as a first set of combined data, the second set of data comprising data that was in files to be unregistered by the merge statement.

8 more elements

A method comprising:

receiving a query plan corresponding to a query, the query plan comprising a tree structure including a set of nodes, each node corresponding to an operation to be performed when executing the query;
executing the query based at least in part on the query plan, the executing comprising:
filtering a first set of files that are to be modified by a merge statement from the set of query statements, the filtering generating a second set of files comprising files from the first set of files that are to be modified by the merge statement;
performing a split operation to send information related to the second set of files to a scan set builder operation in a first portion of the query plan and scan back operation in a second portion of the query plan, the first portion comprising a first branch of the tree structure, the second portion comprising a second branch of the tree structure, each branch including a different set of nodes corresponding to different operations performed when executing the query;
performing the scan set builder operation in the first branch of the tree structure to remove the second set of files from the first set of files, the scan set builder operation generating a third set of files, the third set of files comprising candidate files for consolidation and defragmentation from at least one source table;
performing, using at least a table scan operation, the consolidation and defragmentation of the third set of files, the table scan operation generating a first set of data from scanning the third set of files, the consolidation and defragmentation of the third set of files reducing fragmentation of the source table and reducing an overhead with managing and storing metadata associated with the third set of files, and the scan set builder operation corresponding to a first node below a second node corresponding to the table scan operation in the tree structure corresponding to the query plan;
and
performing a first union all operation to combine the first set of data with a second set of data as a first set of combined data, the second set of data comprising data that was in files to be unregistered by the merge statement.

5 more elements

A non-transitory computer-storage medium comprising instructions that, when executed by one or more processors of a machine, configure the machine to perform operations comprising:

receiving a query plan corresponding to a query, the query including a set of query statements, the query plan comprising a tree structure including a set of nodes, each node corresponding to an operation to be performed when executing the query;
executing the query based at least in part on the query plan, the executing comprising:
filtering a first set of files that are to be modified by a merge statement, the filtering generating a second set of files comprising files from the first set of files that are to be modified by the merge statement;
performing a split operation to send information related to the second set of files to a scan set builder operation in a first portion of the query plan and scan back operation in a second portion of the query plan, the first portion comprising a first branch of the tree structure, the second portion comprising a second branch of the tree structure, each branch including a different set of nodes corresponding to different operations performed when executing the query;
performing the scan set builder operation in the first branch of the tree structure to remove the second set of files from the first set of files, the scan set builder operation generating a third set of files, the third set of files comprising candidate files for consolidation and defragmentation from a source table that are not to be unregistered by the merge statement;
performing, using at least a table scan operation, the consolidation and defragmentation of the third set of files, the table scan operation generating a first set of data from scanning the third set of files, the consolidation and defragmentation of the third set of files reducing fragmentation of the source table and reducing an overhead with managing and storing metadata associated with the third set of files, and the scan set builder operation corresponding to a first node below a second node corresponding to the table scan operation in the tree structure corresponding to the query plan;
and
performing a first union all operation to combine the first set of data with a second set of data as a first set of combined data, the second set of data comprising data that was in files to be unregistered by the merge statement.

5 more elements

Dependencies are inferred from the claim text.