Shard reorganization based on dimensional description in sharded storage systems

An article of manufacture for processing data in a sharded distributed data storage system, wherein the sharded distributed data storage system stores said data on a plurality of shards on a plurality of storage nodes, said article of manufacture comprising a non-transitory machine readable recordable storage medium containing one or more programs which when executed implement the steps of:

• obtaining a dimensional description for a shard reorganization of said data in said sharded distributed data storage system from a user, wherein said dimensional description for said shard reorganization comprises a semantic description of a meaning of said data;
• and
• reorganizing, using a plurality of burst buffer appliances that interchange reorganized data over an interconnect network between said plurality of burst buffer appliances and wherein a given burst buffer appliance comprises a high-speed memory having a substantially lower access time than a disk storage device, a storage of said data and corresponding metadata on at least two of said storage nodes of said sharded distributed data storage system based on said dimensional description.