System and Method for Indexing Streams Containing Unstructured Text Data

A system for indexing data, comprising:

• block-stores adapted to store data-elements of data-streams;
• data-blocks of the block-stores, the stored data-elements being allocated to the data-blocks;
• events of the data-blocks, the block-allocated data-elements being further allocated to the events of the data-blocks, each of the data-blocks comprising one or more events, each of the events comprising the block-allocated data-elements of a corresponding data-block;
• terms generated by splitting the event-allocated data-elements;
• term frequencies calculated based on the frequency of each term in each of the event;
• block-level term frequency data calculated for the event-allocated data-elements that are stored in a corresponding data-block, the block-level term frequency data being based on the term frequencies;
• and,
• tree index structures generated for the event-allocated data-elements based on the block-level term frequency data, the tree index structures comprising Y-tree index structures, the terms being used in the Y-tree index structures as keys.

A computer-usable medium having computer readable instructions stored thereon for execution by a processor, wherein the instructions on the computer-usable medium are adapted to enable a computing device to:

• receive data-streams, wherein the data-streams comprise data-elements;
• store the data-elements of the received data-streams, wherein the stored data-elements are stored in block-stores;
• allocate the stored data-elements to data-blocks of the block-stores;
• further allocate the block-allocated data-elements to events of the data-blocks, wherein each of the data-blocks comprise one or more events, wherein each of the events comprise the block-allocated data-elements of the corresponding data-block;
• split the event-allocated data-elements into terms;
• calculate term frequencies of each term in each of the event;
• calculate block-level term frequency data for the event-allocated data-elements stored in the corresponding data-block based on the term frequencies;
• and,
• generate tree index structures for the event-allocated data-elements based on the block-level term frequency data, wherein the tree index structures comprise Y-tree index structures, wherein the terms are used in the Y-tree index structures as keys.