Automatically scaling compute resources for heterogeneous workloads

A computer-implemented method comprising:

• obtaining a fleet utilization metric indicating, for a fleet of compute instances executing a software application, utilization of worker pools at compute instances of the fleet of compute instances, wherein the worker pools process requests using one or more computing resources of the compute instances upon which the worker pools are executing;
• determining that the fleet utilization metric exceeds a defined threshold;
• and
• scaling a number of compute instances in the fleet of compute instances based on the fleet utilization metric.

A system comprising:

• a fleet of compute instances executing a software application, the fleet of compute instances implemented by a first one or more electronic devices, the software application including instructions that upon execution periodically cause the software application to:
• obtain worker pool utilization metrics and processed requests metrics, the worker pool utilization metrics indicating utilization of worker pools at compute instances of the fleet of compute instances, and the processed requests metrics indicating a number of requests successfully processed by compute instances of the fleet of compute instances in a past time window;
• and
• send the worker pool utilization metrics and the processed requests metrics to a data monitoring service of a service provider network;
• and
• a scaling process of a hardware virtualization service implemented by a second one or more electronic devices, the scaling process including instructions that upon execution cause the scaling process to:
• obtain the worker pool utilization metrics and processed requests metrics from the data monitoring service;
• obtain a pending requests metric indicating a number of requests awaiting processing by the fleet of compute instances;
• calculate a fleet utilization metric based on the worker pool utilization metrics, calculate a current throughput metric based on the processed requests metric and the fleet utilization metric, and calculate a fleet load metric based on the current throughput metric and the pending requests metric;
• and
• scale a number of compute instances in the fleet of compute instances based on the fleet load metric.