Training and deployment of machine learning models

A method comprising:

• receiving inference data for processing by a machine learning model;
• collecting metrics for one or more GPUs associated with a target computing platform, wherein the metrics comprise at least one of:
• a number of the GPUs operable by the target computing platform, or
• an amount of GPU memory accessible to the GPUs operable by the target computing platform;
• executing, within a container loaded with an inference engine, the inference engine on the one or more GPUs to process the inference data using the machine learning model and generate output data, wherein the inference engine is configured based at least in part on the collected metrics.

A system comprising:

• a memory device;
• and
• one or more processing devices, communicatively coupled to the memory device, to:
• receive inference data for processing by a machine learning model;
• collect metrics for one or more processing units available to process the inference data, wherein the metrics comprise at least one of:
• a number of the available processing units,
• a type of the available processing units, or
• an amount of memory accessible to the available processing units;
• load, based at least in part on the collected metrics, an inference engine within a container, the inference engine comprising executable code for execution of the machine learning model;
• and
• perform inference processing of the inference data using the inference engine within the container to generate output data associated with the inference data.