System and method for random-access manipulation of compacted data files

A system for random-access manipulation of a compacted data file, comprising:

• a computing device comprising a memory, a processor, and a non-volatile data storage device;
• a random access engine comprising a first plurality of programming instructions that, when operating on the processor, cause the computing device to:
• receive a data search query directed to the compacted data file, wherein the compacted file is compacted using a reference codebook;
• organize the reference codebook into a hierarchical representation;
• traverse the hierarchical representation to identify a start codeword corresponding to the beginning of the data search query;
• and
• send the start codeword and a plurality of immediately following codewords from the compacted data file to a decoder;
• and
• an estimator module comprising a second plurality of programming instructions that, when operating on the processor, cause the computing device to:
• estimate a first starting bit location in the compacted data file;
• refine the first starting bit location by:
• determining a plurality of codeword boundaries by performing distinct value estimation on the compacted data file, wherein the distinct value estimates correspond to a codeword boundary;
• and
• determining whether a bit sequence starting at the first starting bit location corresponds to a codeword boundary of the plurality of codeword boundaries and, if not, traversing the hierarchical representation until a codeword boundary is located at a new starting bit.