Load balancing in a distributed processing system

A system comprising:

• a data processing apparatus;
• and
• a data store storing instructions that, when executed by the data processing apparatus, cause the data processing apparatus to perform operations comprising:
• obtaining, by a distributed processing system implemented in the data processing apparatus, the distributed processing system including a plurality of data processors, data units to be processed by the distributed processing system, each data unit having a corresponding timestamp that indicates a time the data unit was created, and wherein the time the data unit was created is prior to a time that the data unit is processed by the data processors;
• distributing the data units among the data processors in the distributed processing system for processing by the data processors;
• monitoring, for each data processor in the distributed processing system, a rate of cursor progress for the data processor based on timestamps of data units that have been processed by the data processor, wherein the rate of cursor progress for each data processor specifies a rate of change of an oldest timestamp of an oldest data unit being processed by the data processor based on the rate of change of the times indicated by the respective timestamps of each oldest data unit processed by the data processor;
• determining a progress threshold for the distributed processing system based on the respective rates of cursor progress processed by the plurality of data processors;
• determining, based on a first rate of cursor progress for a first data processor processed by the plurality of data processors, that the first rate of cursor progress for the first data processor does not meet the progress threshold;
• and
• performing one or more load balancing operations on the distributed processing system in response to determining that the first rate of cursor progress for the first data processor does not meet the progress threshold.

A computer readable medium storing instructions that, when executed by a data processing apparatus, cause the data processing apparatus to perform operations comprising:

• obtaining, by a distributed processing system implemented in the data processing apparatus, the distributed processing system including a plurality of data processors, data units to be processed by the distributed processing system, each data unit having a corresponding timestamp that indicates a time the data unit was created, and wherein the time the data unit was created is prior to a time that the data unit is processed by the data processors;
• distributing the data units among data processors in a distributed processing system for processing by the data processors
• monitoring, for each data processor in the distributed processing system that includes a plurality of data processors, a rate of cursor progress for the data processor based on timestamps of data units that have been processed by the data processor, wherein the rate of cursor progress for each data processor specifies a rate of change of an oldest timestamp of an oldest data unit being processed by the data processor based on the rate of change of the times indicated by the respective timestamps of each oldest data unit processed by the data processor;
• determining a progress threshold for the distributed processing system based on the respective rates of cursor progress processed by the plurality of data processors;
• determining, based on a first rate of cursor progress for a first data processor processed by the plurality of data processors, that the first rate of cursor progress for the first data processor does not meet the progress threshold;
• and
• performing one or more load balancing operations on the distributed processing system in response to determining that the first rate of cursor progress for the first data processor does not meet the progress threshold.