Proxy systems and methods for multiprocessing architectures

An apparatus comprising:

• a proxy computing system;
• a client computing system communicatively coupled to the proxy computing system;
• and
• a set of processing devices preloaded with a neural network model and communicatively coupled to the proxy computing system, wherein:
• the proxy computing system receives an inference request from the client computing system, the inference request comprising a model ID associated with the neural network model and an input tensor;
• the proxy computing system receives a statistics request from the client computing system, the statistics request including the model ID and an average inference time request;
• the proxy computing system accesses a load state of each processing device in the set of processing devices;
• the proxy computing system selects a target processing device from the set based on the load states;
• the proxy computing system transmits the inference request to the target processing device;
• the proxy computing system monitors an execution of the inference request by the target processing device based on the neural network model;
• the proxy computing system receives an inference result generated by the target processing device after executing the inference request;
• the proxy computing system computes the average inference time for the inference request execution based on the monitoring;
• and
• the proxy computing system transmits the inference result and the average inference time to the client computing system.