Systems and methods related to a temporal log structure database

A computer program product, the computer program product being tangibly embodied on a non-transitory computer-readable storage medium storing instructions that when executed cause a processor to perform a process, the instructions comprising instructions to:

• receive at a data collection system, a first request to insert a data record within a database of the data collection system;
• place the data record within a buffer in a main memory of the data collection system;
• define a record data structure and a record index structure associated with the data record;
• store the record data structure and the record index structure within a particular storage chunk of a log file comprising a plurality of storage chunks, including the particular storage chunk, in a storage medium of the database, the storage medium being different than the main memory, wherein the particular storage chunk includes:
• at least one associated coarse index that stores two index values, a low index value representing the lowest value stored in the chunk and a high index value representing the highest value stored in the chunk, and
• at least one associated index structure that includes the record index structure that can be used to retrieve the data record, and
• other data records different than the data record associated with the first request;
• and
• receive a second request to retrieve data records from the data collection system, the request including a first search parameter corresponding to the coarse index;
• examine the coarse indexes of the plurality of storage chunks to identify storage chunks that meet the first search parameter of the second request;
• and
• when a storage chunk from the plurality of storage chunks is identified, examine the record index structure to determine data records responsive to the second request and retrieve the data records responsive to the second request.

A data collection system comprising:

• at least one processor;
• a non-transitory storage medium different than main memory storing a log file for the data collection system, the log file comprising a plurality of storage chunks, wherein each storage chunk includes:
• at least one associated coarse index that stores two index values, a low index value representing the lowest value stored in the chunk and a high index value representing the highest value stored in the chunk,
• at least one associated record index structure that can be used to retrieve data records stored in the chunk, and
• wherein the storage chunk includes a group of stored data records;
• and
• a memory storing instructions that, when executed by the at least one processor, cause the system to perform operations comprising:
• receiving a request to retrieve data records from the log file, the request including a first search parameter corresponding to particular coarse indexes,
• examining the particular coarse indexes of the plurality of storage chunks to identify storage chunks that meet the first search parameter of the request, and
• when a storage chunk from the plurality of storage chunks is identified, examining the record index structure to determine data records from the group of data records responsive to the request and retrieving the data records responsive to the request.