Back to results
US 12,730,777 B2Granted
Continuous ingestion of custom file formats
AssigneeSnowflake Inc.
InventorsTyler Arthur Akidau, Thierry Cruanes, Benoit Dageville, Ganeshan Ramachandran Iyer, Subramanian Muralidhar
Life of this patent
Estimating the term…
Abstract
Techniques for continuous ingestion of files using custom file formats are described. A custom file format may include formats not natively supported by a data system. Unstructured files (e.g., images) may also be considered custom file formats. A custom file format may be set using a user defined table function and scanner options.
Claims20
A method for managing a pool of execution nodes in a multi-tenant data system the method comprising:
- in response to identifying a first file of a natively supported format for ingestion, acquiring a first set of execution nodes from the pool of execution nodes;
- performing, using the first set of execution nodes, ingestion of the first file, wherein the first set of execution nodes remains available for concurrent assignment of tasks for other accounts of the multi-tenant data system;
- in response to identifying a second file of a custom format not being natively supported by the multi-tenant data system for ingestion, acquiring a second set of execution nodes from the pool of execution nodes;
- and
- performing, using the second set of execution nodes, ingestion of the second file using a user-defined function (UDF) to parse the custom format, wherein the second set of execution nodes is pinned during said ingestion of the second file to prevent concurrent assignment of tasks for other accounts of the multi-tenant data system.
2 more elements
A system comprising:
- at least one hardware processor;
- and
- at least one memory storing instructions that, when executed by the at least one hardware processor, cause the at least one hardware processor to perform operations comprising:
- in response to identifying a first file of a natively supported format for ingestion, acquiring a first set of execution nodes from a pool of execution nodes in a multi-tenant data system;
- performing, using the first set of execution nodes, ingestion of the first file, wherein the first set of execution nodes remains available for concurrent assignment of tasks for other accounts of the multi-tenant data system;
- in response to identifying a second file of a custom format not being natively supported by the multi-tenant data system for ingestion, acquiring a second set of execution nodes from the pool of execution nodes;
- and
- performing, using the second set of execution nodes, ingestion of the second file using a user-defined function (UDF) to parse the custom format, wherein the second set of execution nodes is pinned during said ingestion of the second file to prevent concurrent assignment of tasks for other accounts of the multi-tenant data system.
5 more elements
A non-transitory machine-storage medium embodying instructions that, when executed by a machine, cause the machine to perform operations comprising:
- in response to identifying a first file of a natively supported format for ingestion, acquiring a first set of execution nodes from a pool of execution nodes in a multi-tenant data system;
- performing, using the first set of execution nodes, ingestion of the first file, wherein the first set of execution nodes remains available for concurrent assignment of tasks for other accounts of the multi-tenant data system;
- in response to identifying a second file of a custom format not being natively supported by the multi-tenant data system for ingestion, acquiring a second set of execution nodes from the pool of execution nodes;
- and
- performing, using the second set of execution nodes, ingestion of the second file using a user-defined function (UDF) to parse the custom format, wherein the second set of execution nodes is pinned during said ingestion of the second file to prevent concurrent assignment of tasks for other accounts of the multi-tenant data system.
2 more elements
Dependencies are inferred from the claim text.