Schema inference for files
AssigneeSnowflake Inc.
InventorYucan Liu
Life of this patent
Estimating the term…
Abstract
Systems and methods for inferring a schema for a text file are provided. The systems and methods perform operations including: accessing a file comprising a plurality of textual records, each textual record of the plurality of textual records being associated with one or more columns of data; sampling a set of textual records from the plurality of textural records; obtaining a hierarchy comprising a plurality of levels of schema types; determining whether an individual column of the one or more columns of data corresponding to the set of textual records is successfully associated with a first level of the plurality of levels of the schema types and, in response, associating a schema type represented by the first level with the individual column of the one or more columns of data corresponding to the plurality of textual records.
Claims26
A system comprising:
- at least one hardware processor;
- and
- at least one memory storing instructions that cause the at least one hardware processor to execute operations comprising:
- accessing a file comprising a plurality of textual records, each textual record of the plurality of textual records being associated with one or more columns of data;
- sampling a set of textual records from the plurality of textual records;
- obtaining a hierarchy comprising a plurality of levels of schema types;
- determining whether an individual column of the one or more columns of data corresponding to the set of textual records is successfully associated with a first level of the plurality of levels of the schema types;
- in response to determining that the individual column of the one or more columns of data corresponding to the set of textual records is successfully associated with the first level, associating a schema type with the individual column of the one or more columns of data corresponding to the plurality of textual records;
- and
- in response to determining that the individual column of the one or more columns of data corresponding to the set of textual records is unsucessfully associated with the first level;
- selecting a second level of the plurality of levels of the schema types;
- and
- determining whether the individual column of the one or more columns of data corresponding to the set of textual records is successfully associated with the second level of the plurality of levels of the schema types.
A method comprising:
- accessing a file comprising a plurality of textual records, each textual record of the plurality of textual records being associated with one or more columns of data;
- sampling a set of textual records from the plurality of textual records;
- obtaining a hierarchy comprising a plurality of levels of schema types;
- determining whether an individual column of the one or more columns of data corresponding to the set of textual records is successfully associated with a first level of the plurality of levels of the schema types;
- in response to determining that the individual column of the one or more columns of data corresponding to the set of textual records is successfully associated with the first level, associating a schema type with the individual column of the one or more columns of data corresponding to the plurality of textual records;
- and
- in response to determining that the individual column of the one or more columns of data corresponding to the set of textual records is uncussessfully associated with the first level;
- selecting a second level of the plurality of levels of the schema types;
- and
- determining whether the individual column of the one or more columns of data corresponding to the set of textual records is successfully associated with the second level of the plurality of levels of the schema types.
A non-transitory computer-storage medium comprising instructions that, when executed by a processor of a machine, configure the machine to perform operations comprising:
- accessing a file comprising a plurality of textual records, each textual record of the plurality of textual records being associated with one or more columns of data;
- sampling a set of textual records from the plurality of textual records;
- obtaining a hierarchy comprising a plurality of levels of schema types;
- determining whether an individual column of the one or more columns of data corresponding to the set of textual records is successfully associated with a first level of the plurality of levels of the schema types;
- in response to determining that the individual column of the one or more columns of data corresponding to the set of textual records is successfully associated with the first level, associating a schema type with the individual column of the one or more columns of data corresponding to the plurality of textual records;
- and
- in response to determining that the individual column of the one or more columns of data corresponding to the set of textual records is unsuccessfully associaated with the first level;
- selecting a second level of the plurality of levels of the schema types;
- and
- determining whether the individual column of the one or more columns of data corresponding to the set of textual records is successfully associated with the second level of the pluraity of levels of the schema types.
Dependencies are inferred from the claim text.