Dynamic management of data stream processing
AssigneeAmazon Technologies, Inc.
InventorGaurav D. Ghare
Life of this patent
Estimating the term…
Abstract
A stream management system may implement dynamic management of a data stream. Utilization data of different partitions of a data stream may be tracked. When routing a data record received at the stream management system, a partition may be dynamically identified for the data recorded. The data record may then be directed to the identified partition. Other management operations, such as repartitioning the data stream or reassigning resources for processing data records in the data stream may be performed based on the utilization data tracked for the partitions.
Claims20
A multi-tenant, stream management service, comprising:
- a plurality of storage nodes configured to store data records in a plurality of different partitions of a data stream for processing;
- a plurality of computing nodes that implement a plurality of stream processing workers to process data records of the data stream and emit the processed data records to one or more destinations;
- a control plane, configured to:
- track utilization of the different partitions of the data stream;
- assign the different partitions to a respective one of the stream processing workers according to the utilization of the different partitions;
- route a data record received at the multi-tenant, stream management service to one of the different partitions according to the utilization of the different partitions;
- and
- wherein the data record is processed by the stream processing worker assigned to the one partition and emitted to at least one of the one or more destinations.
A method, comprising:
- performing, by one or more computing devices:
- tracking utilization of a plurality of different partitions of a data stream that process a plurality of data records received at a multi-tenant, stream management service for the data stream;
- receiving, at the multi-tenant, stream management service, another data record to ingest into the data stream for processing;
- based, at least in part, on the utilization of the different partitions, identifying one of the different partitions for processing the data record;
- and
- directing, by the multi-tenant, stream management service, the data record to be processed as part of the identified partition.
A non-transitory, computer-readable storage medium, storing program instructions that when executed by one or more computing devices cause the one or more computing devices to implement:
- tracking utilization of allocated resources that process different pluralities of data records;
- routing a data record to the allocated resources of one of the different pluralities of data records according to the utilization of the allocated resources for the different pluralities of records;
- and
- emitting the data record processed at the allocated resources of the one plurality of data records to at least one specified destination.
Dependencies are inferred from the claim text.