System and method for random-access manipulation of compacted data files

A system for random-access manipulation of compacted data files, comprising:

• a computing device comprising a memory, a processor, and a non-volatile data storage device;
• a deconstruction engine comprising a first plurality of programming instructions stored in the memory and operable on the processor, wherein the first plurality of programming instructions, when operating on the processor, cause the computing device to:
• deconstruct a data stream into a plurality of data chunklets;
• encode the data stream using a reference codebook by:
• retrieving a reference code for each chunklet from the reference codebook;
• where there is no reference code for a first chunklet, generating a hash code as a new reference code and storing the first chunklet and its newly-created reference code in the reference codebook;
• and
• storing the reference codes corresponding to the data stream in a compacted data file;
• and
• a random access engine comprising a second plurality of programming instructions stored in the memory and operating on the processor, wherein the second plurality of programming instructions, when operating on the processor, cause the computing device to:
• receive a data search query;
• estimate, using an estimator module, a first starting bit location in the compacted data file;
• refine the first starting bit location by:
• determining whether a bit sequence starting at the first starting bit location corresponds to a codeword boundary and, if not, traversing the reference codebook until a codeword boundary is located at a new starting bit;
• and
• traversing from the new starting bit until a start codeword corresponding to the beginning of the data search query is identified;
• and
• sending the start codeword and a plurality of immediately following codewords from the compacted data file to the reconstruction engine for decoding.