Agent-based data pre-processing and data indexing for efficient data retrieval

#It is estimated that the amount of data worldwide will grow from 0.8 to 164 Zettabytes this decade. As an example, Microsoft's Azure® Data Lake Store (a scalable data storage and analytics service) already holds many exabytes and is growing rapidly. Users seek ways to focus on the finer details they really need, but without getting rid of the original data. This is a non-trivial challenge because a single dataset can be used for answering a multitude of questions. As an example, telemetry (e.g., logs, heartbeat information) from various services are stored and analyzed to support a variety of developer tasks (e.g., monitoring, reporting, debugging). With the monetary cost of downtime ranging from $100 k to millions of dollars per hour, real-time processing and querying of this service data becomes critical.

#This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter.

#Methods, systems, apparatuses, and computer program products are directed to the generation of a global index structure. Agents executing on different data sources locally pre-process data generated thereby and index such data. Each index generated by an agent is formatted as a plurality of index nodes. The index nodes are provided to backend server(s) that maintain the global index structure. The backend server(s) generate the global index structure based on the index nodes received by the data sources. As new index nodes are received by the backend servers, the backend servers merge the newly-received index nodes with the global index structure. Global index structure traversal techniques for retrieving search keys are also described herein. Such traversal techniques advantageously limit the number of index nodes that are required to be searched, thereby greatly increasing the speed at which query results are returned (i.e., the read access time is greatly decreased), while also limiting the processing cycles required to find and return such search keys.

#The accompanying drawings, which are incorporated herein and form part of the specification, illustrate embodiments of the application and, together with the description, further explain the principles of the embodiments and to enable a person skilled in the relevant art(s) to make and use the embodiments.

#FIG. 1 is a block diagram of a system for data ingestion and indexing in accordance with an example embodiment.

#FIGS. 2A-2C depict the generation of a hierarchical index structure in accordance with an example embodiment.

#The features and advantages of the embodiments described herein will become more apparent from the detailed description set forth below when taken in conjunction with the drawings, in which like reference characters identify corresponding elements throughout. In the drawings, like reference numbers generally indicate identical, functionally similar, and/or structurally similar elements. The drawing in which an element first appears is indicated by the leftmost digit(s) in the corresponding reference number.

#The following detailed description discloses numerous example embodiments. The scope of the present patent application is not limited to the disclosed embodiments, but also encompasses combinations of the disclosed embodiments, as well as modifications to the disclosed embodiments.

#References in the specification to “one embodiment,” “an embodiment,” “an example embodiment,” or the like, indicate that the embodiment described may include a particular feature, structure, or characteristic, but every embodiment may not necessarily include the particular feature, structure, or characteristic. Moreover, such phrases are not necessarily referring to the same embodiment. Furthermore, when a particular feature, structure, or characteristic is described in connection with an embodiment, it is submitted that it is within the knowledge of persons skilled in the relevant art(s) to implement such feature, structure, or characteristic in connection with other embodiments whether or not explicitly described.