Systems and methods for efficient neural network deployments

A batch processing system for processing requests involving a neural network model, the system comprising:

• one or more computing devices, in which each computing device comprises:
• at least one processor and a memory device;
• a batch producer component that receives data associated with different requests and dynamically assembles chunks of data from at least two different requests into a batch according to one or more constraints by performing steps comprising:
• for each request of at least some of the different requests, using at least one of the one or more constraints associated with the request to select whether the request should be included in a batch;
• and
• assembling data associated with the selected requests into a batch;
• and
• a compute engine component communicatively coupled to the batch producer, the compute engine component processes the batch in a single thread that orchestrates a plurality of threads to share a burden of loading the neural network model from memory to increase data throughput.

A batch producer comprising:

• non-transitory computer-readable medium or media comprising one or more sequences of instructions which, when executed by at least one processor, causes steps to be performed comprising:
• receiving data associated with requests to be processed by using a neural network model, each of the requests being subject to one or more constraints;
• dynamically assembling at least some of the data into a batch using at least one of the one or more constraints by performing steps comprising:
• for each request of at least some of the requests, using at least one of the one or more constraints associated with the request to select whether the request should be included in a batch;
• and
• assembling data associated with the selected requests into a batch;
• and
• causing the batch to be processed using a single thread that orchestrates a plurality of threads to share a burden of loading the neural network model from memory to increase data throughput.