Optimized merge-sorting of data retrieved from parallel storage units

A stream based storage system in a high capacity network, the system comprising:

• a plurality of storage nodes adapted to provide storage and retrieval of time-based data of one or more data streams in response to a receipt of a time-based data retrieval request associated with the one or nor data streams, each of the one or more data streams comprising a plurality of time-ordered data items having a header comprising two or more timestamps representing a time interval associated with each of the plurality of time-ordered data items;
• a plurality of applications providing a user interface to issue the time-based data retrieval request associated with the one or more data streams;
• and
• one or more session managers communicatively coupled to the plurality of applications and to the plurality of storage nodes, the one or more session managers adapted to:
• dynamically allocate shared resources between at least some of the plurality of applications submitting simultaneous time-based data retrieval requests;
• retrieve data requested by the simultaneous time-based data retrieval requests from one or more of the plurality of storage nodes using timestamp based indexes associated with the one or more data streams in a substantially parallel manner;
• merge-sort the retrieved data by time using the timestamp based indexes associated with the one or more data streams;
• identify data streams associated with the time-based data retrieval requests;
• decompose each of the identified data streams into plurality of stream segments, wherein each of the stream segments represents a contiguous range of bytes that is read from one of the plurality of storage nodes and wherein each of the stream segments has a corresponding time index associated therewith;
• combine the time indexes into a plurality of non-overlapping time blocks, wherein the time block is a data structure describing each time range in which one stream segment overlaps another stream segment;
• estimate the size of data to request from each of the identified data streams based on an analysis of the non-oyerlapping time blocks;
• and
• transmit the sorted data to the plurality of applications based on a delivery rate controlled by each application associated with each time-based data retrieval request wherein each timestamp based index associates each of the plurality of stream segments with a plurality of time-ordered data items contained within each stream segment and further associates each of the plurality of stream segments with a particular storage file.