Parallel log structured file system collective buffering to achieve a compact representation of scientific and/or dimensional data

A computer-readable storage medium storing computer-executable instructions for causing a computing device to perform a method for data storage in a collective parallel processing environment, the method comprising:

• receiving data to be written for a plurality of collective processes within a collective parallel processing environment;
• extracting a data pattern for the data to be written for the plurality of collective processes;
• generating a representation describing the data pattern;
• and
• saving the data and the representation.

A high-performance parallel processing computing environment comprising:

• one or more computing devices comprising processing units and memory;
• the one or more computing devices configured to perform operations comprising:
• receiving data to be written for a plurality of collective processes within the parallel processing computing environment;
• extracting a data pattern for the data to be written for the plurality of collective processes;
• generating a representation describing the data pattern;
• and
• saving the data and the representation, wherein the data and the representation are saved using Parallel Log-structured File System (PLFS).