Methods and apparatus for capture and storage of semantic information with sub-files in a parallel computing system

An apparatus for storing at least one file generated by a distributed application in a parallel computing system, wherein said file comprises a plurality of sub-files, comprising:

• a memory;
• and
• at least one hardware device operatively coupled to the memory and configured to:
• obtain a user specification of semantic information related to said file;
• determine semantically meaningful sub-file boundaries for said plurality of sub-files based on said semantic information;
• provide said semantic information as a data structure description to a data formatting library write function, wherein said semantic information is dependent upon a content of said file;
• and
• store said semantic information related to said file with additional metadata related to said file and with one or more of said sub-files using said determined semantically meaningful sub-file boundaries in one or more storage nodes of said parallel computing system.

A data storage system for storing at least one file generated by a distributed application in a parallel computing system, wherein said file comprises a plurality of sub-files, comprising:

• a hardware processing device for obtaining a user specification of semantic information related to said file;
• determining semantically meaningful sub-file boundaries for said plurality of sub-files based on said semantic information;
• and for providing said semantic information as a data structure description to a data formatting library write function, wherein said semantic information is dependent upon a content of said file;
• and
• a memory for storing said semantic information related to said file with additional metadata related to said file and with one or more of said sub-files using said determined semantically meaningful sub-file boundaries in one or more storage nodes of said parallel computing system.