Resource constraint aware deep learning model optimization for serverless-based inference systems

A system, comprising:

• a memory;
• and
• a processing device operatively coupled to the memory, the processing device to:
• detect, by a webhook controller, an inference serverless function invocation;
• generate an optimized version of the inference serverless function using a deep neural network optimizer;
• and
• replace a storage volume in an init container of the inference serverless function with a new storage volume comprising the optimized version of the inference serverless function.

A non-transitory computer-readable storage medium including instructions that, when executed by a processing device, cause the processing device to:

• detect, by a webhook controller, an inference serverless function invocation;
• determine that the inference serverless function can be optimized;
• generate an optimized version of the inference serverless function using a graph compiler, in response to the determination;
• and
• replace, by the processing device of the webhook controller, a storage volume in an init container of the inference serverless function with a new storage volume comprising the optimized version of the inference serverless function.