Scalable and Cost-Efficient Information Retrieval Architecture for Massive Datasets

A computer-implemented method for maintaining a dynamically updating datastore, the method comprising:

• storing, using a first set of storage media, a first data index that indexes elements of a dataset introduced within a storage period, the first set of storage media having a first latency associated therewith;
• storing, using a second set of storage media, a second data index at least partially overlapping the first data index, the second data index indexing elements of the dataset except those introduced within a portion of the storage period, the portion less than the storage period, the second set of storage media having a second latency associated therewith, the second latency being greater than the first latency;
• and
• periodically recomputing new versions of the second data index to include one or more representations of elements indexed in the first data index, the one or more representations of elements maintained in the first data index and the second data index for an overlapping time period.