Directing queries to nodes of a cluster of a container orchestration platform distributed across a host system and a hardware accelerator of the host system

A non-transitory machine readable medium comprising instructions that when executed cause a system to:

• receive queries at an Application Programming Interface (API) gateway of a cluster of a container orchestration platform, wherein the cluster includes a host system comprising a first processing resource and one or more hardware accelerators comprising a second processing resource, wherein the host system comprising the first processing resource or a hardware accelerator of the one or more hardware accelerators comprising the second processing resource comprises a primary node of the cluster, and wherein the host system and the one or more hardware accelerators comprise a plurality of worker nodes of the cluster;
• determine, by the API gateway, a distribution of the queries among the plurality of worker nodes of the host system and the one or more hardware accelerators based on a queuing model that takes into consideration respective compute capacities of the host system and the one or more hardware accelerators;
• perform auto-scaling to run a quantity of instances of an application in the plurality of worker nodes;
• and
• responsive to receipt of the queries, direct the queries to the plurality of worker nodes according to the distribution for processing, by the plurality of worker nodes, the quantity of instances of the application.

A system comprising:

• a cluster comprising a gateway and a host system comprising a processing resource and a plurality of hardware accelerators, wherein the host system or a hardware accelerator of the plurality of hardware accelerators comprises a primary node of the cluster, and the host system and the plurality of hardware accelerators comprise a plurality of worker nodes of the cluster,
• wherein the gateway comprises a non-transitory storage medium comprising instructions executable at the gateway to:
• receive queries at the gateway;
• and
• determine a distribution of the queries among the plurality of worker nodes of the host system and the plurality of hardware accelerators based on a queuing model that takes into consideration respective compute capacities of the host system and the plurality of hardware accelerators,
• wherein the host system is to perform auto-scaling to run a quantity of instances of an application in the plurality of worker nodes, and
• wherein the queries are directed to the plurality of worker nodes according to the distribution for processing, by the plurality of worker nodes, the quantity of instances of the application.