Data management systems and methods
AssigneeSnowflake Computing, Inc.
InventorsBenoit Dageville, Thierry Cruanes, Marcin Zukowski
Life of this patent
Estimating the term…
Abstract
Example data management systems and methods are described. In one implementation, a method identifies multiple files to process based on a received query and identifies multiple execution nodes available to process the multiple files. The method initially creates multiple scansets, each including a portion of the multiple files, and assigns each scanset to one of the execution nodes based on a file assignment model. The multiple scansets are processed by the multiple execution nodes. If the method determines that a particular execution node has finished processing all files in its assigned scanset, an unprocessed file is reassigned from another execution node to the particular execution node.
Claims15
A method comprising:
- receiving a query directed to a database;
- identifying a plurality of files within the database to process in order to generate a response to the query;
- identifying a plurality of execution nodes available to process the plurality of files;
- creating a plurality of scansets and assigning each scanset to a different node of the plurality of execution nodes based on a file assignment model;
- processing, by the plurality of execution nodes, the plurality of scansets in parallel;
- determining, during the processing, that a first execution node has finished processing all files in its assigned scanset;
- and
- responding to the determining by identifying an unprocessed file within one of the plurality of scansets assigned to a second execution node;
- assigning the unprocessed file to the first execution node to be processed by the first execution node;
- and
- generating the response to the query.
Dependencies are inferred from the claim text.