Structuring and rich debugging of inputs and outputs to large language models

A system comprising:

• a computer readable storage medium having program instructions embodied therewith;
• and
• one or more processors configured to execute the program instructions to cause the system to:
• receive one or more user inputs from a user via a graphical user interface, the one or more user inputs indicating:
• a task to be performed by an LLM;
• one or more tools that are available to the system, which may be accessed by the system in response to tool calls from the LLM;
• and
• an output schema for structuring a format of a response from the LLM;
• access tool information relating to the one or more tools;
• identify one or more example tool operations associated with the one or more tools;
• and
• generate a prompt for the LLM including indications of the one or more tools, the one or more example tool operations, the task to be performed, and the output schema.

Non-transitory computer-readable media including computer-executable instructions that, when executed by a computing system, cause the computing system to perform operations comprising:

• receiving one or more user inputs from a user via a graphical user interface, the one or more user inputs indicating:
• a task to be performed by an LLM;
• one or more tools that are available to the system, which may be accessed by the system in response to tool calls from the LLM;
• and
• an output schema for structuring a format of a response from the LLM;
• accessing tool information relating to the one or more tools;
• identifying one or more example tool operations associated with the one or more tools;
• and
• generating a prompt for the LLM including indications of the one or more tools, the one or more example tool operations, the task to be performed, and the output schema.

A computer-implemented method for managing interactions with a large language model (LLM), the computer-implemented method comprising:

• receiving one or more user inputs from a user via a graphical user interface, the one or more user inputs comprising:
• a task to be performed by an LLM;
• an indication of one or more system tools associated with a system and useful for performing the task, the one or more system tools comprising a query object tool, the query object tool comprising a query object operation defining a query structure;
• and
• an indication of an object type having one or more object parameters, the object type associated with the query object tool, the query object tool configured to perform the query object operation on the object type;
• generating a prompt as input to the LLM, the prompt comprising the one or more user inputs;
• responsive to providing the prompt as input to the LLM, receiving from the LLM an LLM output, the LLM output comprising a tool call structured according to the query structure defined by the query object operation of the query object tool, the tool call configured to identify an object within an external data source;
• responsive to querying the external data source using the tool call, receiving from the external data source a query response comprising an object ID;
• generating a second prompt as input to the LLM, the second prompt comprising the object ID and at least a portion of the prompt or at least a portion of the LLM output;
• and
• responsive to providing the second prompt as input to the LLM, receiving from the LLM a second LLM output, the second LLM output comprising a response to the user associated with the task to be performed.