Load balancer that avoids short circuits

A process, comprising:

• performing by one or more load balancers:
• determining a pool of eligible back-end nodes from a fleet of back-end nodes based at least in part on:
• comparing respective recent response times for each back-end node of the fleet of back-end nodes compared to a short response time threshold for the respective back-end node, wherein the short response time threshold is based at least in part on one or more characteristics of the back-end node;
• and
• excluding back-end nodes having recent response times faster than the respective short response time thresholds from the pool of eligible back-end nodes;
• for requests received by the one or more load balancers from one or more clients:
• selecting, from the pool of eligible back-end nodes and in accordance with a load balancing mechanism, respective back-end nodes to handle the requests, wherein the pool of eligible back-end nodes from which the back-end node is selected excludes back-end nodes having recent response times faster than the respective short response time thresholds;
• and
• sending the requests to the respective selected back-end nodes.

A non-transitory computer-readable medium storing program instructions that are executed by one or more processors to implement a load balancer configured to:

• in response to a request,
• select a back-end node from a fleet of back-end nodes to service the request based at least in part on:
• a historical response time profile;
• and
• a current response time;
• wherein a load balancer mechanism for selecting the back-end node specifies that a back-end node with a current response time faster than a short response time threshold has a lower probability of selection than a back-end node with a current response time greater than the short response time threshold.