Managed service for acquisition, storage and consumption of large-scale data streams

A method, comprising:

• performing, by one or more computing devices:
• determining, for a particular data stream comprising a sequence of a plurality of data records, a set of nodes configurable by one or more control components to perform stream management operations based on one or more policies including a stream partitioning policy for partitioning the particular data stream that comprises the sequence of the plurality of data records into a plurality of partitions;
• providing data records in response to record retrieval requests received via one or more programmatic record retrieval interfaces, wherein the one or more programmatic record retrieval interfaces include a first retrieval interface enabling a non-sequential access pattern and a second retrieval interface enabling a sequential access pattern, and wherein a billing rate associated with a use of the first retrieval interface differs from a billing rate associated with a use of the second retrieval interface;
• and
• generating a client billing amount associated with the particular data stream based at least in part on respective usage count metrics of the plurality of record retrieval interfaces.

A non-transitory computer-accessible storage medium storing program instructions that when executed on one or more processors implement a control node of a multi-tenant stream management service, wherein the control node is configured to:

• receive a request to initialize a particular data stream to be comprised of a sequence of a plurality of data records generated by one or more data producers;
• determine, based at least in part on a partitioning policy for partitioning the particular data stream that comprises the sequence of the plurality of data records into a plurality of partitions, one or more parameters to be used to configure one or more subsystems for the particular data stream, including a record retrieval subsystem, wherein the one or more parameters comprise an initial number of nodes to be instantiated in the record retrieval subsystem;
• identify one or more resources to be used for a particular node of the record retrieval subsystem, wherein the particular node is to be configured to implement a plurality of programmatic record retrieval interfaces, including a first retrieval interface enabling a non-sequential access pattern and a second retrieval interface enabling a sequential access pattern;
• and
• configure the particular node of the record retrieval subsystem using the one or more resources.