System, Method, and Computer Program Product for Processing Inference Requests to Execute a Machine-Learning Model

A computer-implemented method, comprising:

• receiving, with at least one processor, an inference request at a request handler having a first programming language capable of multithreading;
• communicating, with at least one processor, a first message based on the inference request from the request handler to at least one object associated with a machine-learning model, the at least one object having a second programming language different than the first programming language;
• determining, with at least one processor, whether to call at least one external service based on the first message and the at least one object;
• communicating, with at least one processor, a second message from the at least one object to an asynchronous function having the first programming language based on determining whether to call the at least one external service;
• calling, with at least one processor, the at least one external service via the asynchronous function while the at least one object associated with the machine-learning model executes;
• communicating, with at least one processor, a third message based on calling the at least one external service from the asynchronous function to the at least one object associated with the machine-learning model;
• determining, with at least one processor, an inference based on the third message and the at least one object associated with the machine-learning model;
• and
• communicating, with at least one processor, a fourth message from the at least one object to the request handler based on the inference.

A computer program product comprising at least one non-transitory computer-readable medium including program instructions that, when executed by at least one processor, causes the at least one processor to:

• receive an inference request at a request handler having a first programming language capable of multithreading;
• communicate a first message based on the inference request from the request handler to at least one object associated with a machine-learning model, the at least one object having a second programming language different than the first programming language;
• determine whether to call at least one external service based on the first message and the at least one object;
• communicate a second message from the at least one object to an asynchronous function having the first programming language based on determining whether to call the at least one external service;
• call the at least one external service via the asynchronous function while the at least one object associated with the machine-learning model executes;
• communicate a third message based on calling the at least one external service from the asynchronous function to the at least one object associated with the machine-learning model;
• determine an inference based on the third message and the at least one object associated with the machine-learning model;
• and
• communicate a fourth message from the at least one object to the request handler based on the inference.