Systems and methods for continuous batching in large language model inference

A computing system, comprising:

• one or more processors;
• and
• a memory storing computer-executable instructions that, when executed by the one or more processors, are to cause the one or more processors to:
• receive a plurality of requests, each request including a prompt for processing by a foundation model and an indication of one or more processing requirements associated with the request;
• use an estimation function, by a first on-mode agent of a plurality of on-mode agents, to determine that a first request of the plurality of requests in a queue can be processed by a first instance of the foundation model within constraints of the one or more processing requirements associated with the first request and without violating the one or more processing requirements associated with any of a plurality of requests currently being processed by the first instance of the foundation model, wherein each of the on-mode agents is associated with a respective instance of the foundation model;
• and
• in response to the determination using the estimation function, process the first request by the first instance of the foundation model.

A non-transitory, computer-readable medium storing computer-executable instructions

• that, when executed by one or more processors, are to cause the one or more processors to:
• receive a plurality of requests, each request including a prompt for processing by a foundation model and an indication of one or more processing requirements associated with the request;
• use an estimation function, by a first on-mode agent of a plurality of on-mode agents, to determine that a first request of the plurality of requests in a queue can be processed by a first instance of the foundation model within constraints of the one or more processing requirements associated with the first request and without violating the one or more processing requirements associated with any of a plurality of requests currently being processed by the first instance of the foundation model, wherein each of the on-mode agents is associated with a respective instance of the foundation model;
• and
• in response to the determination using the estimation function, process the first request by the first instance of the foundation model.