Systems and methods for managing batched artificial intelligence inference requests

A method comprising:

• receiving a synchronous inference request;
• selecting a compute node for execution of the synchronous inference request;
• determining if the synchronous inference request fills an execution batch of the compute node;
• if the synchronous inference request fills the execution batch, causing the compute node to execute the synchronous inference request in the execution batch;
• and
• if the synchronous inference request fails to fill the execution batch, selecting at least one second inference request for execution, and causing the compute node to execute the synchronous inference request and the at least one second inference request in the execution batch.

An article of manufacture comprising:

• a non-transitory computer-readable medium;
• and
• computer-executable instructions carried on the computer-readable medium, the instructions readable by a processor, the instructions, when read and executed, for causing the processor to:
• receive a synchronous inference request;
• select a compute node for execution of the synchronous inference request;
• determine if the synchronous inference request fills an execution batch of the compute node;
• if the synchronous inference request fills the execution batch, cause the compute node to execute the synchronous inference request in the execution batch;
• and
• if the synchronous inference request fails to fill the execution batch, select at least one second inference request for execution, and cause the compute node to execute the synchronous inference request and the at least one second inference request in the execution batch.