Parallel file processing for efficient data ingestion

A system comprising:

• at least one hardware processor;
• at least one memory storing instructions that cause the at least one hardware processor to perform operations comprising:
• splitting an input file into a multiple chunks;
• assigning each chunk of the multiple chunks to one of multiple worker nodes assigned to processing the input file;
• calculating refined boundaries for the multiple chunks, the refined boundaries defining multiple adjusted chunks, the calculating of the refined boundaries comprising:
• determining, by the multiple worker nodes, summary statistics for the multiple chunks;
• determining, by a coordinator, start positions of the multiple chunks based on the summary statistics;
• and
• determining, by the coordinator, end positions of the multiple chunks based on the start positions of the multiple chunks;
• and
• performing data scan passes on the multiple adjusted chunks based on the refined boundaries to populate one or more data structures for storing information extracted from the input file.

A computer-storage medium comprising instructions that, when executed by one or more processors of a machine, configure the machine to perform operations comprising:

• splitting an input file into a multiple chunks;
• assigning each chunk of the multiple chunks to one of multiple worker nodes assigned to processing the input file;
• calculating refined boundaries for the multiple chunks, the refined boundaries defining multiple adjusted chunks, the calculating of the refined boundaries comprising:
• determining, by the multiple worker nodes, summary statistics for the multiple chunks;
• determining, by a coordinator, start positions of the multiple chunks based on the summary statistics;
• and
• determining, by the coordinator, end positions of the multiple chunks based on the start positions of the multiple chunks;
• and
• performing data scan passes on the multiple adjusted chunks based on the refined boundaries to populate one or more data structures for storing information extracted from the input file.