Parallel processing of large data files on distributed file systems with dynamic workload balancing

A computer hardware system configured to perform parallel processing of files, comprising:

• a hardware processor configured to initiate the following executable operations:
• partitioning a file into a set of blocks;
• recording, in a control table, assignments of individual ones of the set of blocks to individual ones in a set of processing engines;
• providing, in response to a request from an individual one in the set of processing engines, access to the control table to identify a block assigned to the individual one in the set of processing engines;
• and
• dynamically reassigning, based upon a determination that all blocks currently assigned to the individual one in the set of processing engines are exhausted, an available block assigned to another one in the set of processing engines to the individual one in the set of processing engines, wherein
• the set of processing engines are configured to perform parallel processing of the set of blocks.

A computer program product, comprising:

• a hardware storage device having stored therein computer-executable program code for performing parallel processing of files,
• the computer-executable program code, which when executed by a computer hardware system, causes the computer hardware system to perform:
• partitioning a file into a set of blocks;
• recording, in a control table, assignments of individual ones of the set of blocks to individual ones in a set of processing engines;
• providing, in response to a request from an individual one in the set of processing engines, access to the control table to identify a block assigned to the individual one in the set of processing engines;
• and
• dynamically reassigning, based upon a determination that all blocks currently assigned to the individual one in the set of processing engines are exhausted, an available block assigned to another one in the set of processing engines to the individual one in the set of processing engines, wherein
• the set of processing engines are configured to perform parallel processing of the set of blocks.