Generating full metadata from partial distributed metadata

A non-transitory computer readable storage medium for tangibly storing computer program instructions capable of being executed by a computer processor, the computer program instructions defining the steps of:

• receiving data to write to disk, the data comprising a subset of a dataset;
• writing a first portion of the data to disk;
• detecting a split boundary after writing the first portion;
• recording metadata describing the split boundary;
• continuing to write a remaining portion of the data to disk;
• and
• after completing the writing of the data to disk:
• generating a partial metadata file for the data, the partial metadata file including the split boundary, and
• transmitting the partial metadata to a partial metadata collector.

An apparatus comprising:

• a processor;
• a storage medium for tangibly storing thereon program logic for execution by the processor, the stored program logic causing the processor to perform the operations of:
• receiving data to write to disk, the data comprising a subset of a dataset,
• writing a first portion of the data to disk,
• detecting a split boundary after writing the first portion,
• recording metadata describing the split boundary,
• continuing to write a remaining portion of the data to disk, and
• after completing the writing of the data to disk:
• generating a partial metadata file for the data, the partial metadata file including the split boundary, and transmitting the partial metadata to a partial metadata collector.