System and method for storing data streams in a distributed environment

A system for storing data, comprising:

• a plurality of block-stores stored on a plurality of distributed devices, the plurality of distributed devices adapted to communicate via a network, the plurality of block-stores storing data-elements received via a plurality of data-streams from the plurality of distributed devices;
• block-identifiers assigned to data-blocks of the block-stores, each of the block-stores comprising one or more data-blocks, the stored data-elements allocated to the one or more data-blocks of the block-stores, each data-block comprising the stored data-elements of only one of the received plurality of data-streams;
• events of the one or more data-blocks, the block-allocated data-elements further allocated to the events of the data-blocks, each of the data-blocks comprise one or more events;
• and,
• indices generated for the event-allocated data-elements.

A non-transitory computer readable medium having computer readable instructions stored thereon for execution by a processor, wherein the instructions on the non-transitory computer readable medium are adapted to:

• receive a plurality of data-streams, wherein the plurality of data-streams are received from a plurality of distributed devices, wherein the plurality of distributed devices are connected via a network, wherein each one of the plurality of data-streams comprise data-elements;
• store the data-elements of the received plurality of data-streams, wherein the stored data-elements are stored in a plurality of block-stores, wherein the block-stores are stored on the plurality of distributed devices;
• assign block-identifiers to data-blocks of the block-stores, wherein the block-identifiers are globally unique, wherein each of the block-stores comprises one or more data-blocks, wherein the data-blocks of a single data-stream are logically grouped, and wherein each of the data-blocks are read and written as a single unit;
• allocate the stored data-elements to the data-blocks of the block-stores, wherein each data-block comprises the stored data-elements of only one of the received plurality of data-streams;
• further allocate the block-allocated data-elements to events of the data-blocks, wherein each of the data-blocks comprise one or more events, wherein each of the events comprises the block-allocated data-elements of the corresponding data-block;
• and,
• generate indices for the event-allocated data-elements.