Low latency and reduced overhead data storage system and method for sharing multiple storage devices by high performance computing architectures

A method for data migration between compute nodes and data storage devices, comprising:

• (a) establishing at least a first and at least a second drive enclosures;
• operatively coupling a plurality of storage controllers in at least a first set of storage controllers and at least a second set of storage controllers;
• operatively coupling a first group of data storage devices to said at least first set of storage controllers and a second group of data storage devices to said at least second set of storage controllers;
• operatively connecting at least one compute node to said at least first and second sets of storage controllers;
• and
• incorporating said at least first set of storage controllers and said first group of data storage devices connected thereto in said at least first drive enclosure, and incorporating said at least second set of storage controllers and said second group of data storage devices connected thereto in said at least second drive enclosure;
• (b) in each of said first and second groups of data storage devices, separating the data storage devices in a plurality of isolated storage pools;
• (c) in each of said at least first and second sets of data storage controllers, appointing each of the data storage controller in each of said at least first and second sets thereof to a respective storage pool from said plurality of the storage pools of the data storage devices, said appointed data storage controller having an exclusive right to write to said respective storage pool;
• (d) generating, at said at least one compute node, at least one data set including at least a first fragment and at least a second fragment, and issuing a first “write” request for a said at least first fragment, and a second “write” request for said at least second fragment of said at least one data set;
• (e) performing a multi-step selection routine including at least a first phase and at least a second phase,
• selecting a target drive enclosure in said at least first phase through the steps of:
• assigning a Data Key (DK) for at least one of said first and second fragments generated by said at least one compute node,
• building an Enclosure Routing Key ERK={DK, Ei} for each drive enclosure from said plurality thereof,
• computing a hash value for said Enclosure Routing Key of said each drive enclosure,
• forming and sorting a first list of said hash values of said plurality of drive enclosures, and
• selecting, from said first sorted list, said target drive enclosure as a healthy drive enclosure having the highest hash value in said first sorted list;
• and
• in said at least second phase, selecting within said selected target drive enclosure a first target data storage controller for servicing said first “write” request for said at least first fragment of said at least one data set, and selecting a second target data storage controller for servicing said second “write” request for said at least second fragment of said at least one data set;
• (f) receiving, at said first and second selected target data storage controllers, said first and second “write” requests, respectively;
• (g) determining, at said first and second selected target data storage controllers, respectively, first and second respective storage pools from said plurality thereof assigned respectively to said first and second selected target data storage controllers, respectively;
• and
• (h) writing said at least first fragment to said first storage pool, and writing said at least second fragment to said second storage pool in a non-collision fashion.