Automated prompt engineering platform

A non-transitory computer-readable storage medium coupled to one or more processors and having instructions stored thereon which, when executed by the one or more processors, cause the one or more processors to perform operations for engineering of prompt templates for prompting large language models (LLMs), the operations comprising:

• receiving user input defining configuration parameters for a set of prompt templates;
• providing a configuration file responsive to the user input;
• processing the configuration file to generate a set of prompts by populating at least one placeholder of each prompt template with at least one input parameter of input parameters groups defined in the configuration file;
• transmitting the prompts in the set of prompts to one or more LLMs;
• receiving, from the one or more LLMs, a set of outputs;
• transmitting a set of metric evaluation prompts to at least one LLM, each metric evaluation prompt being provided using an evaluation prompt template and an output;
• receiving, from the at least one LLM, a set of evaluation results, each evaluation result corresponding to a respective prompt in the set of prompts;
• and
• responsive to the set of evaluation results, selectively deploying prompt templates in the set of prompt templates for production use in prompting the one or more LLMs.

A system, comprising:

• a computing device;
• and
• a computer-readable storage device coupled to the computing device and having instructions stored thereon which, when executed by the computing device, cause the computing device to perform operations for engineering of prompt templates for prompting large language models (LLMs), the operations comprising:
• receiving user input defining configuration parameters for a set of prompt templates,
• providing a configuration file responsive to the user input,
• processing the configuration file to generate a set of prompts by populating at least one placeholder of each prompt template with at least one input parameter of input parameters groups defined in the configuration file,
• transmitting the prompts in the set of prompts to one or more LLMs,
• receiving, from the one or more LLMs, a set of outputs,
• transmitting a set of metric evaluation prompts to at least one LLM, each metric evaluation prompt being provided using an evaluation prompt template and an output,
• receiving, from the at least one LLM, a set of evaluation results, each evaluation result corresponding to a respective prompt in the set of prompts, and
• responsive to the set of evaluation results, selectively deploying prompt templates in the set of prompt templates for production use in prompting the one or more LLMs.