Distributing application traffic to servers based on dynamic service response time

A method for distributing application traffic, the method comprising:

• relaying, by a service gateway, a first service request for a first service session from the service gateway to a server of a plurality of servers, the first service request being received from a host and being associated with a service request time;
• receiving, by the service gateway, from the server, a service response, the service response being associated with a service response time;
• calculating, by the service gateway, a service processing time for the first service request based on the service request time and the service response time;
• comparing, by the service gateway, the service processing time with an expected service processing time for the server stored in a datastore;
• receiving, by the service gateway, from the host, a second service request for a second service session;
• and
• selectively relaying, by the service gateway, the second server request to the server based on the service processing time.

A system for distributing application traffic, the system comprising:

• a server of a plurality of servers, the server being configured to process service requests;
• and
• a service gateway comprising a processor and a computer readable storage medium having computer readable program code embodied therewith, wherein when the computer readable program code is executed by the processor, causes the service gateway to:
• relay a first service request for a first service session from the service gateway to the server, the first service request being received from a host and being associated with a service request time;
• receive, from the server, a service response, the service response being associated with a service response time;
• calculate a service processing time for the first service request based on the service request time and the service response time;
• compare the service processing time with an expected service processing time for the server stored in a datastore to determine whether the service processing time exceeds the expected service processing time by at least a threshold amount, wherein the expected service processing time is based at least in part on a service attribute of the first service request or a service attribute of the server;
• update a server busy indicator for the server in response to the comparing, wherein a server busy indicator for each of the plurality of servers is maintained at the service gateway, wherein the updating the server busy indicator for the server comprises:
• in response to determining that the service processing time exceeds the expected service processing time by at least the threshold amount, updating, by the service gateway, the server busy indicator to indicate that the server is busy;
• and
• in response to determining that the service processing time does not exceed the expected service processing time by at least the threshold amount, updating, by the service gateway, the server busy indicator to indicate that the server is not busy;
• receive, from the host, a second service request for a second service session;
• and
• selectively relay the second server request to the server based on the service processing time, wherein the selectively relaying the second server request to the server comprises:
• checking, by the service gateway, the server busy indicator for the server;
• in response to determining that the server busy indicator indicates that the server is busy, placing, by the service gateway, the second service request in a service request buffer of the service gateway and maintaining a connection to the host;
• and
• in response to determining that the server busy indicator indicates that the server is not busy, relaying, by the service gateway, the second service request from the service gateway to the server.