System and method for data compaction utilizing distributed codebook encoding

A system for data compaction utilizing distributed codebook encoding, comprising:

• a computing device comprising a processor, a memory, and a non-volatile data storage device;
• a library engine comprising a first plurality of programming instructions stored in the memory which, when operating on the processor, causes the computing device to:
• receive digital data to be compacted using a codebook, from a source computing device;
• add new sourceblocks to a codebook based on received non-training data to be compacted;
• create a behavior codebook from a set of rules, limitations, policies, or other configuration or behavioral settings for a library engine;
• wherein the behavior codebook alters or determines the specific operations of the data compaction using a codebook;
• encode and decode data using the codebook and behavior codebook;
• return the newly encoded or decoded data to the source computing device;
• and
• allow the distribution of the codebook to other devices for decoding of data;
• wherein the codebook may be used to decode data, but may not be used to encode data properly without the behavior codebook;
• wherein the behavior codebook comprises:
• behavioral rules that control and prioritize which portions of the source data are encoded using specific codewords from the codebook;
• configuration parameters that set limits on sourceblock sizes and types that can be compacted;
• and
• encoding policies that determine specific operations of the data compaction process;
• wherein the behavior codebook is maintained separately from but linked to the codebook, such that the behavior codebook controls how the codebook is utilized during encoding while the codebook maintains the actual codewords for encoding and decoding.