Annotating datasets without redundant copying

A non-transitory computer-readable storage medium for tangibly storing computer program instructions capable of being executed by a computer processor, the computer program instructions defining the steps of:

• reading a raw dataset, the raw dataset comprising a first set of columns and a first set of rows;
• generating an annotation dataset, the annotation dataset comprising a second set of columns and a second set of rows;
• assigning row identifiers to each row in the second set of rows, the row identifiers aligning the second set of rows with the first set of rows based on the underlying storage of the raw dataset and annotation dataset;
• and
• writing the annotation dataset to a distributed storage medium.

An apparatus comprising:

• a processor;
• and
• a storage medium for tangibly storing thereon program logic for execution by the processor, the stored program logic causing the processor to perform the operations of:
• reading a raw dataset, the raw dataset comprising a first set of columns and a first set of rows;
• generating an annotation dataset, the annotation dataset comprising a second set of columns and a second set of rows;
• assigning row identifiers to each row in the second set of rows, the row identifiers aligning the second set of rows with the first set of rows based on the underlying storage of the raw dataset and annotation dataset;
• and
• writing the annotation dataset to a distributed storage medium.