Monitoring and steering service requests to acceleration components

A system comprising:

• a set of acceleration components;
• a host, configured to send messages, corresponding to a service, to any of a set of acceleration components;
• and
• a load-balancer configured to:
• (1) track an input buffer occupancy corresponding to each of the set of acceleration components, (2) upon a first input buffer occupancy of a first acceleration component from among the set of the acceleration components meeting a predetermined criterion, generate a first penalty value for the first acceleration component based at least on the first input buffer occupancy associated with the first acceleration component, and compare a second value indicative of an availability of the second acceleration component to receive messages from the host to a second dynamically updated value to determine whether the host can send a next message to the second acceleration component.

A method in a system comprising a host configured to send messages, corresponding to a service, to any of a set of acceleration components, the method comprising:

• tracking an input buffer occupancy and a processing rate corresponding to each of the set of acceleration components;
• the host initiating a transmission of a first message to a first acceleration component selected from among the set of acceleration components as long as, based on a comparison of a first value indicative of an availability of the first acceleration component with a first dynamically updated value, the first acceleration component is determined to be in a healthy state;
• and
• upon a first input buffer occupancy of the first acceleration component meeting a predetermined criterion, generating a first penalty value for the first acceleration component based at least on the first input buffer occupancy and a first processing rate associated with the first acceleration component, and comparing a second value indicative of an availability of a second acceleration component to receive messages from the host to a second dynamically updated value to determine whether the host can send a next message to the second acceleration component.