De-multiplexing streaming data from a fast write ahead durable log into segments
AssigneeEMC IP Holding Company LLC
InventorsAndrei Paduroiu, Srikanth Satyanarayana
Life of this patent
Estimating the term…
Abstract
Implementations are provided herein for managing streaming data that appended and multiplexed into a durable write ahead replicated log. By writing data in the durable log, large amounts of small writes can be processed quickly. Data in the durable log can be de-multiplexed and packaged into segment containers. Segment containers can serialize stream segment specific data and can be stored in long term storage. Data that has been stored in long term storage can be truncated from the durable write ahead log making room for new data.
Claims20
An apparatus configured to a memory and at least one processor configured to:
- receive one or more data streams, wherein each data stream includes one or more data portions;
- monitor one or more storage parameters, wherein each storage parameter corresponds to one or more of:
- data input load of each data stream, data write rate of each stream, number of events in each stream, and data ingestion rates of one or more storage devices;
- and
- elastically scale data storage resources based on any changes to the at least one monitored storage parameter.
A method comprising:
- receiving one or more data streams, wherein each data stream includes one or more data portions;
- monitoring one or more storage parameters, wherein each storage parameter corresponds to one or more of:
- data input load of each data stream, data write rate of each stream, number of events in each stream, and data ingestion rates of one or more storage devices;
- and
- elastically scaling data storage resources based on any changes to the at least one monitored storage parameter.
Dependencies are inferred from the claim text.