System for organizing and fast searching of massive amounts of data

A process carried out in a server having a query request processor coupled to a non-relational database file system via an access manager computer program executing on said server, said access manager computer program functioning to control said server to receive performance metric data and configuration data and event data gathered by probes coupled to resources being monitored, convert said performance metric data into Unicode and store at least said encoded performance metric data and configuration data separately in files in said non-relational database file system and store said event data, said server coupled to receive web requests containing search queries via a web request controller wherein each search query can specify one or more searches on one or more levels of nesting, said web request controller forwarding web requests to a query request processor which controls said server to execute searches specified in each said search query, said server also coupled to a result cache where at least performance metric data and/or configuration data and/or event data located as a result of processing said search queries by said query request processor is stored, and wherein said query request processor is structured to control said server to execute each said search query by searching said configuration data and/or said encoded performance metric data and/or said event data specified in each search in a search query using a a filter or matching criteria expressed in a regular expression where all regular expressions have a syntax controlled by rules which must be followed, said process carried out by said query request processor comprising:

• A) parsing each said search query and using information in one or more searches in said search query and said regular expressions, to determine relevant data to access from said non-relational database file system, and retrieving all relevant data from instances of resources having a resource type matching a resource type specified in said search query, and wherein each said search query can specify one or more searches on each of one or more levels of nesting;
• B) creating a process thread to process said retrieved relevant data accessed in step A for each different instance of a resource having a said resource type specified in a first search on a first level of nesting specified in said search query, where there is one thread per resource instance, and wherein said server is further programmed to maintain an application properties file which contains a field configured to store a number that defines how many threads are created;
• C) in each process thread, applying all filters specified in said search query for said first level of nesting sequentially against said relevant retrieved data being processed in said thread to determine if said resource instance being processed in said thread qualifies, and wherein, if more than one search is specified in said search query on said first level, each said filter or matching criteria of a search specified in said search query which is not the first search on said first nesting level is applied only to relevant configuration or performance metric data or event data specified in said search being processed which is not said first search on said first nesting level from the instance of said resource type specified in said first search of said first nesting level which qualified in the preceding search, where qualifying means the relevant data met the criteria specified in said filter or matching condition specified in a regular expression of said previous search, and, if said relevant data of a resource instance does not qualify any search's filter, discarding the relevant non qualifying data from said resource instance being processed by said process thread;
• D) in each thread whose resource instance had relevant data which qualified all searches on said first nesting level in step C, determining if there are any sub paths to another nesting level where sub resources related to said resource type specified in a first search on said first nesting level of said search query will have relevant data specified in searches for said another nesting level searched, and, if so, accessing relevant data specified in a first search on said another nesting level from a sub resource instance specified by a first search on said another nesting level which is related to the resource instance being processed in said process thread of a resource type specified in said first search on said first nesting level of said search query and which was qualified by a last search on said first nesting level, and applying all said filters specified in said searches specified in said search query for said another nesting level sequentially against said relevant data of said instances of said related sub resource, each said search on said another nesting level being applied only to relevant data of said related sub resource instance which qualified in the previous search on said another nesting level being processed so as to determine if said instance of said sub resource qualified all searches on said another nesting level, and discarding said relevant data said sub resource instance did not qualify any of said searches on said another nesting level being processed, and, if an instance of said related sub resource qualified all searches on said another nesting level, determining if any more sub paths to another nesting level are specified in said search query, and repeating the processing specified herein of accessing relevant performance and/or configuration and/or event data for an instance of said sub resource which is related to the instance of said sub resource specified in said first search on the preceding nesting level which qualified all searches on said preceding nesting level, and applying all filters of searches specified in said search query to be carried out on said another nesting level being processed against it seriatem sequentially in the manner described above until there are no more sub paths to process;
• E) in each thread, moving said instance of said resource type specified in said first search of said first level to said results cache along with all data which qualified all said searches on said first level, if said instance of said resource type specified in said first search at said first nesting level qualified all searches on said first nesting level, and, for each nesting level below said first level, moving said instance of said sub resource specified in a first search at said nesting level to said results cache if said instance of said sub resource qualified all searches on said nesting level, and moving to said result cache all relevant data of said instance of said sub resource specified in said first search at said nesting level which passed all said searches on said nesting level.

A data storage apparatus having a non-relational database file system in memory of a server in which time-series performance metric data is stored by an access manager computer program running on said server, wherein said non-relational database file system is hierarchical and has one directory for each day and a subdirectory for each instance of a resource type from which said time-series performance metric data was gathered, said access manager computer program structured to control said server to create said hierarchical non-relational database file system by performing the following steps:

• A) receiving time-series performance metric data from one or more probes structured or programmed to monitor attributes of instances of various resource types of a monitored computer system, said one or more probes also monitoring event data and configuration data of said instances of various resource types, said one or more probes gathering, for each instance of a resource type, for each attribute, a performance metric data measurement during each of a plurality of timeslots which span an entire day so as to generate a time-series of performance metric data numerical values, and encoding each said performance metric data numerical value into a Unicode character so as to generate for each said time-series of performance metric data numerical values a corresponding Unicode time-series;
• and
• B) storing each said Unicode time-series in said non-relational database file system in a file as a time-slice of Unicode characters that represent said time-series of performance metric data numerical values collected from an attribute of an instance of a resource type during said timeslots which span one day, each said file stored in a subdirectory devoted to storing attribute data collected from a particular instance of a resource type, said subdirectory within said directory devoted to storing data gathered on one day, said directory having in its name the date of collection of each said time series of performance metric data numerical values represented by a said Unicode time-series, and wherein the name of each said subdirectory identifies said resource instance from which said time series of performance metric data numerical values represented by said Unicode time series stored in said file was collected, said name of said subdirectory serving as an index of said instance of said resource type;
• and wherein said data storage apparatus is structured to store said Unicode time-series using a storage method which creates said hierarchical file system so that the structure of said hierarchical file system encodes metadata of each said Unicode character in a Unicode time-series, said metadata comprising date, timeslot number, attribute identity and resource identity of each said Unicode character, said storage method effectively creating a logical array of performance metric data indexed by date of collection on one axis and indexed by said identity of said resource instance on another axis;
• and wherein said data storage apparatus is further structured to perform, in the storing of each said time-series of Unicode characters, a pattern matching search based on a known or frequently used search query data pattern, and further structured to create a reference to a storage location of each said Unicode time-series which represents a time-series of performance metric data numerical values which includes data which matched said known or frequently used search query data pattern, and wherein said data storage apparatus is further structured to simultaneously store said reference for each said Unicode time series which represents a time-series of performance metric data numerical values which includes data which met said known or frequently used query data pattern along with said Unicode time-series thereby speeding up searches for said known or frequently used query data pattern.

A data storage apparatus having a non-relational database file system in memory of a server in which multiple Unicode time-series are stored, each Unicode time-series representing a time-series of performance metric data numerical values, each said Unicode time series being created and stored by an access manager computer program running on said server, said access manager computer program structured to control said server to:

• A) receive time-series data from a probe structured or programmed to monitor one or more attributes of one or more instances of one or more resource types of one or more monitored computer systems, virtual servers or other computer processes, said probe structured to transmit a time-series of performance metric data numerical values for each attribute of each instance of each resource type being monitored, said access manager computer program structured to control said server to encode each said time-series of performance metric data numerical values into a times-series of Unicode characters;
• and
• B) store each said time-series of Unicode characters as a time-slice of Unicode characters using a storage method which creates a hierarchical data structure which classifies on one axis each said time-series of Unicode characters by the date on which said time-series of performance metric data numerical values represented by said time-series of Unicode characters was collected, and said storage method classifying each said time-series of Unicode characters by the identity of said instance of said resource type from which said time-series of performance metric data numerical values represented by said time series of Unicode characters was collected on another axis, and wherein said access manager computer program is further structured to, in storing each said time-series of Unicode characters, create a key based upon the time slot of collection of each said performance metric data numerical value in a time-series of performance metric data numerical values and represented by a Unicode character in each said time-series of Unicode characters, and to further create an identifier of the particular attribute of the particular instance of a resource type from which was collected each said time-series of performance metric data numerical values represented by a time-series of Unicode characters, and wherein said data storage apparatus is structured to store said key and said identifier in association with each said time series of Unicode characters.