Distribution of batch data to sharded readers

A system configured to route sharded data to an application of a plurality of applications, the system comprising:

• a concentrator including computer-executable instructions that, when executed, operate to:
• obtain a first pointer generated by a naming service;
• retrieve a sorted batch of data from a storage service based at least in part on the first pointer, the data including records sorted by hash keys;
• combine the sorted batch of data with at least one other sorted batch of data to create grouped batch data;
• send the grouped batch data to the storage service for at least temporary storage;
• sub-partition the grouped batch data into a plurality of groups each associated with a different key range that is a predetermined division of possible hash keys, wherein a first number of key ranges of the hash keys is greater than a second number of the plurality of applications;
• and
• send second pointers that locate at least first records of the grouped batch data, stored by the storage service, to a first shard data stream of a plurality of shard data streams, the second pointers being sent based at least in part on the first records being within a first key range of the key ranges of the hash keys by which the records of the sorted batch of data are sorted, the first shard data stream to convey the first records to an application of the plurality of applications.

A system configured to route data shards to a plurality of applications, the system comprising:

• a naming service to:
• identify a metric name from data provided by an incoming data source;
• generate a hash key for the metric name to create a record;
• and
• generate a sorted batch of data that includes the record and other records, the record and the other records each including corresponding hash keys of a plurality of hash keys;
• a storage service to at least temporarily store the sorted batch of data;
• a data stream to transport a first pointer that locates the sorted batch of data stored by the storage service;
• a concentrator in communication with the data stream to:
• obtain the first pointer via the data stream;
• retrieve the sorted batch of data;
• combine the sorted batch of data with at least one other sorted batch of data based at least in part on the corresponding hash keys to create grouped batch data;
• send the grouped batch data to the storage service for at least temporary storage;
• sort records in the grouped batch data into a plurality of sub-partitions based at least in part on the corresponding hash keys in the record and the other records, each sub-partition associated with a different key range that is a predetermined division of possible hash keys of the plurality of hash keys, wherein a first number of key ranges into which the records in the grouped batch data are sorted is greater than a second number of the plurality of applications to which the data shards are routed;
• and
• send, to a first shard data stream of a plurality of shard data streams, a second pointer and a third pointer that locate at least first records of the grouped batch data, stored by the storage service, based at least in part on the first records being within a first key range of a subset of the plurality of hash keys associated with the sorted batch of data located via the first pointer;
• the first shard data stream to transport the second pointer and the third pointer to a first application of the plurality of applications.