Autonomous simulated testing and benchmarking framework for agentic AI systems

A system, comprising:

• at least one processor;
• and
• a memory storing computer-executable instructions that, when executed by the at least one processor, cause the system to:
• bind a simulation agent to tool-access interfaces of an agentic AI system configured to invoke external tools;
• intercept a request emitted at the tool-access interfaces;
• generate a protocol-compliant response using synthetic data and a simulated environment;
• substitute the protocol-compliant response for an expected response from the external tools, thereby enabling the agentic AI system to execute tasks without invoking the external tools;
• execute a healthy run and a fault-inserted run of a predefined task suite;
• and
• generate a performance vector comprising task-completion rate, accuracy, efficiency, resilience, and fault-recovery, based on the healthy run and the fault-inserted run.

A non-transitory computer-readable storage medium storing instructions which, when executed by at least one processor, cause the at least one processor to perform operations comprising:

• binding a simulation agent to at least one tool-access interface of an agentic AI system to replace at least one external tool;
• intercepting, by the simulation agent, a request emitted at the at least one tool-access interface;
• generating, by the simulation agent, a protocol-compliant response using synthetic data and a simulated environment;
• substituting, by the simulation agent, the protocol-compliant response for an expected response from the at least one external tool, thereby enabling the agentic AI system to execute tasks without invoking the at least one external tool;
• executing, by the agentic AI system, a healthy run and a fault-inserted run of a predefined task suite;
• and
• generating, by the simulation agent, a performance vector comprising task-completion rate, accuracy, efficiency, resilience, and fault-recovery, based on the healthy run and the fault-inserted run.