System, Method, and Computer Program Product for Dynamically Processing Model Inference or Training Requests

A computer-implemented method comprising:

• receiving, with at least one processor, a plurality of requests from a plurality of requesting systems;
• creating, with at least one processor, a plurality of instantiations of at least one machine learning model based on the plurality of requests and service data associated with each requesting system of the plurality of requesting systems;
• streaming, with at least one processor, data associated with at least one inference request of the plurality of requests to each instantiation of the plurality of instantiations;
• adjusting, with at least one processor, a rate limit for each instantiation of the plurality of instantiations based on the service data associated with at least one requesting system associated with the instantiation, resulting in an adjusted rate limit;
• and
• processing, with at least one processor, at least one request of the plurality of requests with an instantiation of the plurality of instantiations based on the adjusted rate limit.

A computer program product comprising at least one non-transitory computer-readable medium including program instructions that, when executed by at least one processor, causes the at least one processor to:

• receive a plurality of requests from a plurality of requesting systems;
• create a plurality of instantiations of at least one machine learning model based on the plurality of requests and service data associated with each requesting system of the plurality of requesting systems;
• stream data associated with at least one inference request of the plurality of requests to each instantiation of the plurality of instantiations;
• adjust a rate limit for each instantiation of the plurality of instantiations based on the service data associated with at least one requesting system associated with the instantiation, resulting in an adjusted rate limit;
• and
• process at least one request of the plurality of requests with an instantiation of the plurality of instantiations based on the adjusted rate limit.