System and method for fast parallel data processing in distributed storage systems

A distributed storage system for fast parallel data processing, the system comprising:

• an electronic memory;
• and
• a hardware processor couple to the electronic memory and configured to:
• receive at least one digital object;
• determine whether the at least one digital object has a fixed data block structure;
• in response to determining that the at least one digital object has a fixed data block structure, determine a size of each fixed data block of the at least one digital object;
• determine a number of archive chunks for storing the at least one digital object on a selected plurality of storage servers based at least upon the size of each fixed data block;
• and
• partition the at least one digital object into the number of archive chunks for storing on the selected plurality of storage servers using erasure coding.

A non-transitory computer readable medium storing computer executable instructions for fast parallel data processing in a distributed storage system, the instructions being configured for:

• receiving at least one digital object;
• determining, by a processor of a computing device, whether the at least one digital object has a fixed data block structure;
• in response to determining that the at least one digital object has a fixed data block structure, determining by the processor a size of each fixed data block of the at least one digital object;
• determining a number of archive chunks for storing the at least one digital object on a selected plurality of storage servers based at least upon the size of each fixed data block;
• and
• partitioning the at least one digital object into the number of archive chunks for storing on the selected plurality of storage servers using erasure coding.