System and method for efficient multi-stage querying of archived data

A system for searching indexed packages, comprising:

• a processor;
• memory coupled to the processor, the memory comprising computer executable instructions that, when executed by the processor, perform a method for searching indexed packages;
• comprising:
• ingesting records of data, the records of data ingested in packages, the ingesting comprising:
• indexing the records of data based on a parameter, the indexing including storing an index file in each of the packages, the index file in each of the packages indexing records of data in that package using the parameter, wherein the parameter is characterized by a range of values;
• generating indexed packages for the records of data based on the parameter, wherein each of the generated indexed packages corresponds to one of a plurality of subsets of the range of values;
• generating metadata for the indexed packages, the metadata for each indexed package comprising a reference to that indexed package of records and a package key that characterizes that indexed package of records based on a value of the parameter, wherein the package key differs from the parameter;
• and
• storing the indexed packages in a data repository;
• querying the records of data based on a query defining a search value of the parameter, the querying comprising:
• searching the metadata based on the search value of the parameter, wherein the search value is defined within one of the subsets;
• identifying a package key for the metadata referencing the search value of the parameter;
• loading, from a file-based cache, an indexed package based on the identified package key, when the indexed package is stored in the cache;
• and
• loading, from the data repository which is an archive storage, the indexed package when the indexed package is not stored in the cache.

A computer program product comprising a non-transitory computer readable medium storing instructions executable by a processor to search indexed packages in a computing environment, the set of operations comprising:

• ingesting records of data, the records of data ingested in packages, the ingesting comprising:
• indexing the records of data based on a parameter, the indexing including storing an index file in each of the packages, the index file in each of the packages indexing records of data in that package using the parameter, wherein the parameter is characterized by a range of values;
• generating indexed packages for the records of data based on the parameter, wherein each of the generated indexed packages corresponds to one of a plurality of subsets of the range of values;
• generating metadata for the indexed packages, the metadata for each indexed package comprising a reference to that indexed package of records and a package key that characterizes that indexed package of records based on a value of the parameter, wherein the package key differs from the parameter;
• and
• storing the indexed packages in a data repository;
• querying the records of data based on a query defining a search value of the parameter, the querying comprising:
• searching the metadata based on the search value of the parameter, wherein the search value is defined within one of the subsets;
• identifying a package key for the metadata referencing the search value of the parameter;
• loading, from a file-based cache, an indexed package based on the identified package key, when the indexed package is stored in the cache;
• and
• loading, from the data repository which is an archive storage, the indexed package when the indexed package is not stored in the cache.