Using reinforcement learning to scale queue-based services

A system comprising:

• one or more processors;
• and
• a memory having stored thereon instructions that, upon execution by the one or more processors, cause the one or more processors to perform operations comprising:
• accessing historical data comprising, for a point in time:
• a compute capacity indicating a number of allocated compute instances;
• and
• usage metrics indicating pending processing requests in a queue and a current utilization of available compute instances;
• determining a compute scaling adjustment by applying a machine learning model to the compute capacity and to the usage metrics, the compute scaling adjustment indicating an adjustment to a number of compute instances;
• computing a reward value by:
• calculating a negative penalty based on the compute scaling adjustment;
• normalizing and negating a number of tasks in the queue,
• weighing the number of compute instances by a current load, wherein the current load is a proportion of a current number of compute instances that is used by tasks in the queue, and
• calculating the reward value from the negative penalty, the normalized and negated number of tasks, and the weighted number of compute instances;
• and
• providing the reward value to the machine learning model, wherein the machine learning model adjusts one or more internal parameters to maximize a cumulative reward.

A computer program product tangibly embodied in a non-transitory machine-readable storage medium, including instructions configured to cause one or more data processors to perform actions including:

• accessing historical data comprising, for a point in time:
• a compute capacity indicating a number of allocated compute instances;
• and
• usage metrics indicating pending processing requests in a queue and a current utilization of available compute instances;
• determining a compute scaling adjustment by applying a machine learning model to the compute capacity and to the usage metrics, the compute scaling adjustment indicating an adjustment to a number of compute instances;
• computing a reward value by:
• calculating a negative penalty based on the compute scaling adjustment;
• normalizing and negating a number of tasks in the queue,
• weighing the number of compute instances by a current load, wherein the current load is a proportion of a current number of compute instances that is used by tasks in the queue, and
• calculating the reward value from the negative penalty, the normalized and negated number of tasks, and the weighted number of compute instances;
• and
• providing the reward value to the machine learning model, wherein the machine learning model adjusts one or more internal parameters to maximize a cumulative reward.