Supervisor routine for large language model output

A non-transitory computer-readable medium storing computer-executable instructions that, when executed, cause a computing system to perform operations comprising:

• accessing, by an online system, an agentic workflow, the agentic workflow comprising a set of nodes, the set of nodes comprising a plurality of prompt nodes and a plurality of agentic nodes, wherein each prompt node comprises computer-executable instructions for prompting a large language model to generate an output for the agentic workflow, wherein each agentic node comprises computer-executable instructions for interfacing with a computing system, and wherein the plurality of prompt nodes comprise a supervisor node that comprises computer-executable instructions for prompting the large language model to apply guidelines to an output of one of the plurality of prompt nodes;
• receiving, by the online system, a first set of natural-language text from a client device associated with a user, wherein the first set of natural-language text relates to an action to be performed by the online system for the user;
• executing a prompt node of the set of nodes that is connected to the supervisor node through an edge in the agentic workflow, wherein executing the prompt node comprises:
• accessing the computer-executable instructions of the prompt node, the computer-executable instructions including a first prompt template for generating a first prompt to the large language model;
• generating the first prompt for the large language model based on the first prompt template of the prompt node and the first set of natural-language text;
• inputting the first prompt to the large language model;
• receiving a first output from the large language model;
• responsive to receiving the first output from the large language model, executing the supervisor node by:
• accessing primary instructions of the supervisor node, the primary instructions including a second prompt template, wherein the second prompt template including a instructions for a large language model to generate a set of error scores, each guideline score representative of a likelihood that the first output includes a type of error of a set of types of errors, wherein the instructions to generate the set of error scores comprises text instructions for how to evaluate an output of a large language model to determine whether an error of a corresponding type is present in the output;
• generating a second prompt for the large language model based on the second prompt template and the first output;
• inputting the second prompt to the large language model;
• and
• receiving a second output from the large language model, wherein the second output comprises the set of error scores;
• comparing each error score to a threshold associated with a respective type of error;
• and
• in response to at least one error score exceeding a respective threshold, re-executing the prompt node.