Managing a computing cluster using durability level indicators

Software stored in a non-transitory form on a computer-readable medium, managing data in a distributed data processing system including a plurality of processing nodes, the software including instructions for causing a computing system to:

• maintain a plurality of data stores in the system,
• each processing node of the plurality of processing nodes maintaining a disjoint subset of the data stores,
• each data store of the plurality of data stores having a durability level of a plurality of durability levels, at least some processing nodes each maintaining data stores at multiple different durability levels, the plurality of durability levels including a first durability level and a second durability level with a relatively greater degree of durability than the first durability level,
• maintain a plurality of time indicators for the processing system, each time indicator corresponding to a respective different durability level, including maintaining a first indicator corresponding to the first durability level;
• process a plurality of sets of data units using two or more of the plurality of processing nodes, including
• assigning each data unit a time interval of a plurality of successive time intervals and assigning a first set of data units to a given time interval;
• and
• storing data units in data stores at multiple levels of durability, including storing at least some data units in data stores having the first durability level and storing at least some data units in data stores having the second durability level;
• wherein maintaining the time indicators includes updating the first time indicator to represent a time value such that the data units assigned a time interval represented by the time value are stored in a data store having at least the first durability level.

An apparatus for managing data in a distributed data processing system, the apparatus including:

• a plurality of data stores in the distributed data processing system, where each processing node of the plurality of processing nodes maintains a disjoint subset of the data stores,
• each data store of the plurality of data stores having a durability level of a plurality of durability levels, at least some processing nodes each maintaining data stores at multiple different durability levels, the plurality of durability levels including a first durability level and a second durability level with a relatively greater degree of durability than the first durability level,
• a plurality of time indicators for the processing system, each time indicator corresponding to a respective different durability level, including maintaining a first indicator corresponding to the first durability level;
• one or more processors for processing a plurality of sets of data units using two or more of the plurality of processing nodes, including
• assigning each data unit a time interval of a plurality of successive time intervals and assigning a first set of data units to a given time interval;
• and
• storing data units in data stores at multiple levels of durability, including storing at least some data units in data stores having the first durability level and storing at least some data units in data stores having the second durability level;
• wherein maintaining the time indicators includes updating the first time indicator to represent a time value such that the data units assigned a time interval represented by the time value are stored in a data store having at least the first durability level.

A computing system for managing data in a distributed data processing system including a plurality of processing nodes, the computing system including:

• means for maintaining a plurality of data stores in the system,
• each data store of the plurality of data stores having a durability level of a plurality of durability levels, at least some processing nodes each maintaining data stores at multiple different durability levels, the plurality of durability levels including a first durability level and a second durability level with a relatively greater degree of durability than the first durability level,
• means for maintaining a plurality of time indicators for the processing system, each time indicator corresponding to a respective different durability level, including maintaining a first indicator corresponding to the first durability level;
• means for processing a plurality of sets of data units using two or more of the plurality of processing nodes, including
• assigning each data unit a time interval of a plurality of successive time intervals and assigning a first set of data units to a given time interval;
• and
• storing data units in data stores at multiple levels of durability, including storing at least some data units in data stores having the first durability level and storing at least some data units in data stores having the second durability level;
• wherein maintaining the time indicators includes updating the first time indicator to represent a time value such that the data units assigned a time interval represented by the time value are stored in a data store having at least the first durability level.