PID controller for event ingestion throttling

A device, comprising:

• a memory;
• and
• a processor, the processor configured to execute instructions stored in the memory to:
• process accepted events, wherein each accepted event is associated with a respective ingested timestamp, a respective processing-start timestamp, and a respective processing-complete timestamp, and wherein events are accepted at a rate of a target rate limit;
• obtain a current error value as a difference between a target lag time and a second value, the target lag time indicating a configured maximum time within which an accepted event is to be processed, and the second value being an average of differences between the respective processing-complete timestamps and the respective ingested timestamps;
• obtain a base throttle based on previous error values and the current error value, wherein the base throttle is a rate that is less than the target rate limit and indicates a maximum number of events per a unit of time to be accepted for processing;
• and
• accept subsequent events at a new rate obtained from the base throttle.

A non-transitory computer-readable storage medium, comprising executable instructions that, when executed by a processor, perform operations, comprising:

• processing accepted events, wherein each accepted event is associated with a respective ingested timestamp, a respective processing-start timestamp, and a respective processing-complete timestamp, and wherein events are accepted at a rate of a target rate limit;
• obtaining a current error value as a difference between a target lag time and a second value, the target lag time indicating a configured maximum time within which an accepted event is to be processed, and the second value being an average of differences between the respective processing-complete timestamps and the respective ingested timestamps;
• obtaining a base throttle based on previous error values and the current error value, wherein the base throttle is a rate that is less than the target rate limit and indicates a maximum number of events per a unit of time to be accepted for processing;
• obtaining a control value as an active processing time divided by a sum of the active processing time with an average active processing time;
• scaling, using the control value, the base throttle to obtain a new rate;
• and
• accepting subsequent events at the new rate obtained from the base throttle.