Metrics data storage and access

A computer-implemented method for data management, the method being executed by one or more processors and comprising:

• receiving data, wherein the data comprises a plurality of fields;
• transforming the data of each type into a byte array using mapping tables, wherein the byte array is an encoded format of the data, where each field of the data is encoded as a field ID and a field type ID that are short integer variables;
• merging and storing the byte arrays of multiple data into a binary file, wherein the binary file comprises multiple blocks with each block comprising multiple byte arrays;
• generating indexes for common fields of different data in the binary file;
• receiving a retrieval request requesting records including a common field of a particular value;
• determining storage locations of one or more records satisfying the retrieval request using the index of the common field, the index of a common field being included in i) an offset table comprising a position offset of each matching record in the binary file, and ii) a key table indicating a number of matching records for each particular value and a stating location of the number of records in the offset table;
• and
• obtaining the one or more records satisfying the retrieval request from the binary file using the corresponding storage locations.

A non-transitory computer-readable storage medium coupled to one or more processors and having instructions stored thereon which, when executed by the one or more processors, cause the one or more processors to perform operations for data management, the operations comprising:

• receiving data, wherein the data comprises a plurality of fields;
• transforming the data of each type into a byte array using mapping tables, wherein the byte array is an encoded format of the data, where each field of the data is encoded as a field ID and a field type ID that are short integer variables;
• merging and storing the byte arrays of multiple data into a binary file, wherein the binary file comprises multiple blocks with each block comprising multiple byte arrays;
• generating indexes for common fields of different data in the binary file;
• receiving a retrieval request requesting records including a common field of a particular value;
• determining storage locations of one or more records satisfying the retrieval request using the index of the common field, the index of a common field being included in i) an offset table comprising a position offset of each matching record in the binary file, and ii) a key table indicating a number of matching records for each particular value and a stating location of the number of records in the offset table;
• and
• obtaining the one or more records satisfying the retrieval request from the binary file using the corresponding storage locations.

A system, comprising:

• a computing device;
• and
• a computer-readable storage device coupled to the computing device and having instructions stored thereon which, when executed by the computing device, cause the computing device to perform operations for data management, the operations comprising:
• receiving data, wherein the data comprises a plurality of fields;
• transforming the data of each type into a byte array using mapping tables, wherein the byte array is an encoded format of the data, where each field of the data is encoded as a field ID and a field type ID that are short integer variables;
• merging and storing the byte arrays of multiple data into a binary file, wherein the binary file comprises multiple blocks with each block comprising multiple byte arrays;
• generating indexes for common fields of different data in the binary file;
• receiving a retrieval request requesting records including a common field of a particular value;
• determining storage locations of one or more records satisfying the retrieval request using the index of the common field, the index of a common field being included in i) an offset table comprising a position offset of each matching record in the binary file, and ii) a key table indicating a number of matching records for each particular value and a stating location of the number of records in the offset table;
• and
• obtaining the one or more records satisfying the retrieval request from the binary file using the corresponding storage locations.