System and method for data compression with intrusion detection

A system for data compaction with intrusion detection, comprising:

• a computing device comprising a processor and a memory;
• an intrusion detection module comprising a first plurality of programming instructions stored in the memory which, when operating on the processor, causes the computing device to:
• receive a codeword data stream;
• use one or more algorithms to compute a probability distribution of a plurality of codewords within the codeword data stream;
• compare the computed probability distribution with a reference probability distribution to compute an amount of divergence between the computed probability distribution and the reference probability distribution;
• and
• when the computed amount of divergence exceeds a configured risk sensitivity threshold:
• store the computed divergence, the computed probability distribution, and the codeword as anomalous event data in a database;
• generate an intrusion alert, the intrusion alert comprising the anomalous event data;
• and
• send the intrusion alert to a user interface to be viewed by a user;
• and
• a codebook training module comprising a second plurality of programming instructions stored in the memory which, when operating on the processor, causes the computing device to:
• receive a training dataset;
• use the training dataset to create the reference probability distribution;
• send the reference probability distribution to the intrusion detection module;
• receive data;
• format the received data into a test dataset;
• retrieve a first measured probability distribution associated with the previous training dataset from a monitor database;
• use one or more algorithms to measure a second probability distribution of the test dataset;
• compare the first and second measured probability distributions to compute the difference in distribution statistics between the test dataset and the previous training dataset;
• in response to the difference in distribution statistics' exceeding a pre-determined difference threshold:
• use the test dataset to retrain encoding and decoding algorithms;
• utilize the retrained algorithms to create new data sourceblocks;
• create new codeword for each new data sourceblock;
• store each new data sourceblock and its associated new codeword in an updated codebook;
• and
• send the updated codebook to a plurality of encoding and decoding machines.