System and method for reduction of data transmission by optimization of inference accuracy thresholds

A non-transitory machine-readable medium having instructions stored therein, which when executed by a processor, cause the processor to perform operations for aggregating data in a data aggregator of a distributed environment using data collected by a data collector of the distributed environment, the data collecting being remote to the data aggregator, the operations comprising:

• obtaining, by the data aggregator, a plurality of synthetic data sets;
• obtaining an acceptable error level for a downstream consumer of the aggregated data using the plurality of synthetic data sets;
• utilizing the acceptable error level as a threshold for inference accuracy, the threshold being associated with the downstream consumer;
• obtaining an inference model based on the threshold;
• distributing the inference model to a data collector;
• obtaining a reduced data size representation of the data collected by the data collector;
• and
• reconstructing the data collected by the data collector using the reduced data size representation of the data and an inference generated by the inference model to obtain the aggregated data, the reconstructed data being different from the data collected by the data collector by less than the acceptable error level.

A data aggregator, comprising:

• a processor;
• and
• a memory coupled to the processor to store instructions, which when executed by the processor, cause the processor to perform operations for aggregating data in a data aggregator of a distributed environment using data collected by a data collector of the distributed environment, the data collecting being remote to the data aggregator, the operations comprising:
• obtaining, by the data aggregator, a plurality of synthetic data sets;
• obtaining an acceptable error level for a downstream consumer of the aggregated data using the plurality of synthetic data sets;
• utilizing the acceptable error level as a threshold for inference accuracy, the threshold being associated with the downstream consumer;
• obtaining an inference model based on the threshold;
• distributing the inference model to a data collector;
• obtaining a reduced data size representation of the data collected by the data collector;
• and
• reconstructing the data collected by the data collector using the reduced data size representation of the data and an inference generated by the inference model to obtain the aggregated data, the reconstructed data being different from the data collected by the data collector by less than the acceptable error level.