Managing a distributed system processing a publisher's streaming data

A system comprising:

• a processor;
• and
• a computer readable storage medium storing processor-executable computer program instructions, the computer program instructions comprising instructions for managing a distributed system processing streaming data received over a network from a data source, the distributed system comprising a plurality of data stores and a plurality of workers, wherein each of the plurality of data stores is dedicated to the data source, and wherein each of the plurality of workers is dedicated to a respective data store, the method comprising:
• selecting, by each of the workers, based on a criteria, a sampling ratio, the sampling ratio defining a fraction of data items selected for storage in the respective data store, wherein a first worker selects a first sampling ratio and a second worker selects a second sampling ratio, and wherein the first and second sampling ratios are different;
• responsive to receiving, by the distributed system, streaming data comprising data items from the data source:
• assigning, by the distributed system, data items from the streaming data to the workers;
• and
• storing, by each of the assigned workers, based on the respective sampling ratios of each of the assigned workers, data items from the assigned data items in the respective data store;
• responsive to receiving, by the distributed system, a query request, wherein the query request comprises a constraint:
• distributing, by the distributed system, the query request to each of the workers;
• processing, by each of the workers, the query request, wherein the processing comprises:
• determining a tally of data items stored in the respective data store meeting the constraint of the query;
• and
• estimating a count of data items assigned to the worker meeting the constraint of the query by dividing the tally by the respective sampling ratio of the respective worker;
• estimating, by the distributed system, a total of data items received by the distributed system from the data source meeting the constraint of the query by totaling the estimated counts of data items from each of the workers;
• and
• providing, by the distributed system, the estimated total of data items received by the distributed system from the data source meeting the criteria.

A non-transitory computer-readable medium with computer executable instructions executed by a processor to perform a method of managing a distributed system processing streaming data received over a network from a data source, the distributed system comprising a plurality of data stores and a plurality of workers, wherein each of the plurality of data stores is dedicated to the data source, and wherein each of the plurality of workers is dedicated to a respective data store, the method comprising:

• selecting, by each of the workers, based on a criteria, a sampling ratio, the sampling ratio defining a fraction of data items selected for storage in the respective data store, wherein a first worker selects a first sampling ratio and a second worker selects a second sampling ratio, and wherein the first and second sampling ratios are different;
• responsive to receiving, by the distributed system, streaming data comprising data items from the data source:
• assigning, by the distributed system, data items from the streaming data to the workers;
• and
• storing, by each of the assigned workers, based on the respective sampling ratios of each of the assigned workers, data items from the assigned data items in the respective data store;
• responsive to receiving, by the distributed system, a query request, wherein the query request comprises a constraint:
• distributing, by the distributed system, the query request to each of the workers;
• processing, by each of the workers, the query request, wherein the processing comprises:
• determining a tally of data items stored in the respective data store meeting the constraint of the query;
• and
• estimating a count of data items assigned to the worker meeting the constraint of the query by dividing the tally by the respective sampling ratio of the respective worker;
• estimating, by the distributed system, a total of data items received by the distributed system from the data source meeting the constraint of the query by totaling the estimated counts of data items from each of the workers;
• and
• providing, by the distributed system, the estimated total of data items received by the distributed system from the data source meeting the criteria.