System and method for data storage, transfer, synchronization, and security using automated model monitoring and training

A system for storing, retrieving, and transmitting data in a highly compact format, comprising:

• a computing device comprising a processor and a memory;
• a monitor database stored in the memory of the computing device, wherein the monitor database comprises a previous training dataset, a first estimated probability distribution associated with the previous training dataset, performance metrics, and model predictions;
• a codebook training module comprising a first plurality of programming instructions stored in the memory and operable on the processor, wherein the second plurality of programming instructions, when operating on the processor, cause the processor to:
• receive data;
• send the received data to a compression engine to be compressed, if it is image data;
• format the received data into a test dataset;
• retrieve the first estimated probability distribution associated with the previous training dataset from the monitor database;
• use one or more algorithms to estimate a second probability distribution of a run-time dataset;
• compare the first estimate and the second estimate to determine a difference value, wherein the difference value exceeds a difference threshold value, use the test dataset to retrain encoding and decoding algorithms;
• utilize the retrained algorithms to create new data sourceblocks;
• create a new codeword for each new data sourceblock;
• store each new data sourceblock and its associated new codeword in an updated codebook;
• and
• the compression engine comprising a second plurality of programming instructions stored in the memory and operable on the processor, wherein the third plurality of programming instructions, when operating on the processor, cause the processor to:
• receive image data to be compressed;
• apply a mathematical transform on the image data to be compressed, such as a discrete cosine transform;
• applies an entropy encoding algorithm to the quantized and transformed data, such as Huffman coding, Shannon coding, or others;
• and
• sends the encoded, quantized, and transformed data to a codebook training module;
• wherein the compression engine performs only steps useful for lossless compression, without loss of data fidelity or content.