Automatic latency optimization for cpu-based dnn serving

A method of optimizing thread allocation for a model serving system, the method comprising:

• estimating a batch size for inference requests of the model serving system using a thread optimization system of the model serving system, the model serving system serving a deep learning model and including a plurality of threads corresponding to a total thread count for processing inferences for the deep learning model;
• automatically determining, using an optimizer component of the thread optimizer system, a first optimal configuration that defines a number of inference instances, a number of threads per inference instance, and a sub-batch size per inference instance for processing a batch of inference requests of the batch size using intra-operator parallelism with the plurality of threads that minimizes average per-patch latency, the first optimal configuration being determined with reference to a plurality of predetermined model profiles that define single-inference average batch latencies for different combinations of thread counts and batch sizes, the predetermined model profiles being used as input to a dynamic programming algorithm that identifies optimal configurations that minimize the average per-batch latency;
• and
• allocating compute resources based on the number of inference instances, the number of threads per inference instance, and the sub-batch size per inference instance indicated by the first optimal configuration.

A non-transitory computer readable medium on which are stored instructions that, when executed, cause a programmable device to perform functions of:

• estimating a batch size for inference requests of a model serving system using a thread optimization system of the model serving system, the model serving system serving a deep learning model and including a plurality of threads corresponding to a total thread count for processing inferences for the deep learning model;
• automatically determining, using an optimizer component of the thread optimizer system, a first optimal configuration that defines a number of inference instances, a number of threads per inference instance, and a sub-batch size per inference instance for processing a batch of inference requests of the batch size using intra-operator parallelism with the plurality of threads that minimizes average per-patch latency, the first optimal configuration being determined with reference to a plurality of predetermined model profiles that define single-inference average batch latencies for different combinations of thread counts and batch sizes, the predetermined model profiles being used as input to a dynamic programming algorithm that identifies optimal configurations that minimize the average per-batch latency;
• and
• allocating compute resources based on the number of inference instances, the number of threads per inference instance, and the sub-batch size per inference instance indicated by the first optimal configuration.