Energy efficiency for inferencing workloads

A non-transitory machine-readable medium have first instructions stored thereon configured to be execution on a multi-core processor of a compute platform having a plurality of processor cores, wherein execution of the first instructions enable the compute platform to:

• generate metrics relating to core utilization and one or more performance metrics while running an inference large language model (LLM) on the compute platform via execution of second instructions on a portion of the plurality of cores;
• compare the one or more performance metrics with one or more target performance metrics;
• and
• adjust the core utilization for running the inference LLM to meet the one or more performance metrics.

A compute platform, comprising:

• a processor having a plurality of cores;
• memory, operatively coupled to the processor;
• instructions, loaded in the memory or stored in a storage device operatively coupled to the processor and configured to be executed on processor cores among the plurality of processor cores to enable the compute platform to:
• run an inference large language model (LLM) on the compute platform via execution of a portion of the instructions on a portion of the plurality of cores;
• generate metrics relating to core utilization for running the inference LLM and one or more performance metrics;
• compare the one or more performance metrics with one or more target performance metrics;
• and
• adjust the core utilization for running the inference LLM to meet the one or more performance metrics.