Pid controller for event ingestion throttling

A system, comprising:

• a memory;
• and
• a processor, the processor configured to execute instructions stored in the memory to:
• obtain a base throttle indicating a maximum number of events per a unit of time to be accepted for processing, the events corresponding to a plurality of event sources;
• for each event source of the plurality of event sources, obtain a respective control value based on a processing time of events corresponding to the event source relative to an average processing time of the events;
• for each event source, scale the base throttle using the respective control value and a respective scaling function to obtain a respective scaled throttled rate limit, wherein a first event source having a processing time that is greater than the average processing time is assigned a first scaled throttled rate limit that is less than the base throttle, and a second event source having a processing time that is less than the average processing time is assigned a second scaled throttled rate limit that is greater than the first scaled throttled rate limit;
• and
• accept, for processing, subsequent events corresponding to each event source according to the respective scaled throttled rate limit.

A non-transitory computer-readable storage medium comprising instructions that, when executed by a processor, perform operations, the operations comprising:

• obtaining a base throttle indicating a maximum number of events per a unit of time to be accepted for processing, the events corresponding to a plurality of event sources;
• for each event source of the plurality of event sources, obtaining a respective control value based on a processing time of events corresponding to the event source relative to an average processing time of the events;
• for each event source, scaling the base throttle using the respective control value and a respective scaling function to obtain a respective scaled throttled rate limit, wherein a first event source having a processing time that is greater than the average processing time is assigned a first scaled throttled rate limit that is less than the base throttle, and a second event source having a processing time that is less than the average processing time is assigned a second scaled throttled rate limit that is greater than the first scaled throttled rate limit;
• and
• accepting, for processing, subsequent events corresponding to each event source according to the respective scaled throttled rate limit.