Parallel checksumming of data chunks of a shared data object using a log-structured file system

A compute node apparatus in a parallel computing system for executing a Log-Structured File System client that stores a data chunk generated by said parallel computing system to a shared data object on a storage node in said parallel computing system, said compute node apparatus comprising:

• a memory;
• and
• at least one hardware device operatively coupled to the memory and configured to:
• process said data chunk generated by said parallel computing system using said Log-Structured File System client, wherein said Log-Structured File System client executes on one or more of said compute node and said burst buffer node in said parallel computing system;
• determine, by said Log-Structured File System client, a checksum value for said data chunk as said data chunk is written by said Log-Structured File System client to said storage node, wherein a plurality of other Log-Structured File System clients determine a plurality of checksums for a corresponding plurality of other data chunks of said shared data object in parallel;
• and
• provide, by said Log-Structured File System client, said checksum value with said data chunk to said storage node for storage as part of said shared object, wherein said storage node stores said data chunk with said corresponding checksum value as part of said shared object with said plurality of checksums for said corresponding plurality of other data chunks of said shared data object, and wherein said shared object is shared by a plurality of distributed processes in said parallel computing system.

A Log-Structured File System server apparatus in a parallel computing system for storing a data chunk as a shared object, said storage node apparatus comprising:

• a memory;
• and
• at least one hardware device operatively coupled to the memory and configured to:
• receive, by said Log-Structured File System server, said data chunk with a corresponding checksum value from a Log-Structured File System client on a compute node in said parallel computing system, wherein said data chunk is generated by said parallel computing system using Log-Structured File System client, and wherein said corresponding checksum value is generated by said Log-Structured File System client as said data chunk is written by said Log-Structured File System Client, wherein a plurality of other log-structured file system clients determine a plurality of checksums for a corresponding plurality of other data chunks of said shared data object in parallel;
• and
• store, by said Log-Structured File System server, said received data chunk and said corresponding checksum value to a shared data object on said storage node, wherein said storage node stores said data chunk with said corresponding checksum value as part of said shared object with said plurality of checksums for said corresponding plurality of other data chunks of said shared data object, and wherein said shared object is shared by a plurality of distributed processes in said parallel computing system.