Efficient processing of time series data

A method for processing data, comprising:

• receiving time series data associated with a plurality of metrics;
• writing said time series data to a first file in a spool format during a first time period, wherein the time series data is organized in the first file into blocks of data, each block of data storing time series data received during a sample period of a plurality of sample periods, said spool format includes a set of blocks of information, each block of information stores a time sample for a set of metrics;
• converting said time series data from said spool format in the first file to a second format in a second file suited for long term storage after said first time period, said second format includes a header and a footer, wherein the time series data is organized in the second file into blocks of data, each block of data storing time series data for a single metric received over the plurality of sample periods including records that are individually compressed, the footer containing a record for each block of data identifying the metric associated with the block of data and a pointer indicting the start of the block of data corresponding to the metric, the footer includes pointers to each block of data;
• receiving a query of said time series data associated with a particular metric after said first time period;
• and
• performing said query of said time series data after said first time period by searching the footer of the second file for the particular metric and following a pointer associated with the particular metric to the block of data containing said time series data in said second format associated with the particular metric.

One or more processor readable storage devices storing processor readable code for programming one or more processors, said processor readable code comprising:

• a writer, said writer receives data about a set of metrics associated with a software application and stores said data in blocks of data in a first file in a first format during a first time period, each block of data contains a beginning sample time and all data acquired from the beginning sample time through a sample period, wherein each instance of data stored in a block of data is associated with a metric;
• a converter, said converter accesses said data in said first format and converts said data from said first format to a second format in a second file having a header and blocks of data after the first time period, the header includes a start time indicating the earliest sample time associated with the data stored in the blocks of data, a stop time indicating the latest sample time associated with the data stored in the blocks of data and a sample period for each instance of data stored in a block of data based on the beginning sample times contained in the blocks of data in the first format, each block of data stores all data about a single metric acquired between the start sample time and the stop sample time indicated in the header, said second file has a footer that includes pointers to each block of data;
• and
• a manager that receives a first query of said data during said first time period, performs said first query using said first file during said first time period, receives a second query of said data after said first time period, and performs said second query of said data in said second file, said performing said second query includes reading said footer to identify the block of data containing the data associated with the metric and following a pointer to a corresponding block of data associated with the metric as well as reading the corresponding block of data and reporting based on information in said corresponding block.