Back to results
US 9,753,928 B1Granted
System and method for identifying delimiters in a computer file
AssigneeTrifacta Inc.
InventorsAaron J. Elmore, Adam E. Silberstein, Joseph M. Hellerstein, Sean Kandel
Life of this patent
Estimating the term…
Abstract
A system and method automatically identifies any or all of potential row, column and string delimiters in a file in which such delimiters are unknown to the program making such identification.
Claims18
A computer-implemented method of parsing a file, comprising:
- receiving a plurality of tokenizing schemas, each tokenizing schema specifying a plurality of delimiters;
- for each of the plurality of tokenizing schemas:
- generating a tokenized version of the file based on the tokenizing schema, the tokenized version of the file obtained by replacing characters of the file with tokens representing delimiters according to the tokenized schema;
- for each of a plurality of sizes, identifying in the tokenized version of the file, a pattern of delimiters that repeats in the tokenized version of file, the pattern having the size;
- and
- for each of the plurality of patterns of delimiters identified, determining a score based on a frequency with which the pattern of delimiters appears in the tokenized version of the file;
- selecting a tokenizing schema from the plurality of tokenized schemas based on the scores of the plurality of patterns of delimiters associated with each of the plurality of tokenized schemas;
- and
- parsing the file using the selected tokenizing schema.
6 more elements
A computer system for parsing a file, comprising:
- a computer processor;
- and
- a non-transitory computer readable storage medium storing instructions for:
- receiving a plurality of tokenizing schemas, each tokenizing schema specifying a plurality of delimiters;
- for each of the plurality of tokenizing schemas:
- generating a tokenized version of the file based on the tokenizing schema, the tokenized version of the file obtained by replacing characters of the file with tokens representing delimiters according to the tokenized schema;
- for each of a plurality of sizes, identifying in the tokenized version of the file a pattern of delimiters that repeats in the tokenized version of file, the pattern having the size;
- and
- for each of the plurality of patterns of delimiters identified, determining a score based on a frequency with which the pattern of delimiters appears in the tokenized version of the file;
- selecting a tokenizing schema from the plurality of tokenized schemas based on the scores of the plurality of patterns of delimiters associated with each of the plurality of tokenized schemas;
- and
- parsing the file using the selected tokenizing schema.
9 more elements
A non-transitory computer readable storage medium storing instructions for parsing a file, wherein the instructions are for:
- receiving a plurality of tokenizing schemas, each tokenizing schema specifying a plurality of delimiters;
- for each of the plurality of tokenizing schemas:
- generating a tokenized version of the file based on the tokenizing schema, the tokenized version of the file obtained by replacing characters of the file with tokens representing delimiters according to the tokenized schema;
- for each of a plurality of sizes, identifying in the tokenized version of the file, a pattern of delimiters that repeats in the tokenized version of file, the pattern having the size;
- and
- for each of the plurality of patterns of delimiters identified, determining a score based on a frequency with which the pattern of delimiters appears in the tokenized version of the file;
- selecting a tokenizing schema from the plurality of tokenized schemas based on the scores of the plurality of patterns of delimiters associated with each of the plurality of tokenized schemas;
- and
- parsing the file using the selected tokenizing schema.
6 more elements
Dependencies are inferred from the claim text.