Generation of diversified validation test suite for generative artificial intelligence powered tools

A method for generating a diversified validation test suite for generative artificial intelligence (AI) powered tools, the method comprising:

• executing programmatic control logic stored in memory of a controller having a processor, a memory, and input/output (I/O) ports, the programmatic control logic comprising a diversified validation test suite application (DVTSA) for a generative AI powered tool or large language model (LLM) tool;
• receiving a seed test input or user input via the I/O ports;
• analyzing the seed test input or user input;
• extracting key elements within the seed test input or user input for variations;
• performing a coverage measurement of outputs of the analyzing and extracting steps;
• and
• evaluating outputs relative to predefined coverage metrics and selectively and continuously iterating to increase LLM tool input robustness and output robustness from a first level to a second level greater than the first level, in both production and pre-production LLM tool processes, wherein the method progressively reduces computational resource utilization, progressively increases computational efficiency, and progressively reduces reliance on a human validator.

A method for generating a diversified validation test suite for generative artificial intelligence (AI) powered tools, the method comprising:

• executing programmatic control logic stored in memory of a controller having a processor, the memory, and input/output (I/O) ports, the processor executing the programmatic control logic, the programmatic control logic comprising a diversified validation test suite application (DVTSA) for a generative AI powered tool or large language model (LLM) tool, the DVTSA including control logic for:
• receiving a seed test input or user input via the I/O ports, analyzes the seed test input or user input and extracts key elements for variations, including:
• extracting syntactical information from the seed test input or user input, including:
• collecting all seed test input or user input coverage terms including extracting individual key terms;
• quantifying output of coverage points based on corresponding test outputs and language information relating to extracted individual key terms;
• automatically generating a script utilizing the individual key terms extracted;
• and
• utilizing supplementary documents and the quantified output of the coverage points to collect all coverage points, wherein the supplementary documents further comprise:
• predefined language information including linguistic databases, mathematical databases, syntactic and semantic databases, such as:
• GridXML, ROBOT Script, natural language databases, mathematical terminology and databases of variables, statements, and terms defined with respect to hardware;
• wherein the input coverage terms further comprise:
• predefined term and statements for known inputs and outputs of the method;
• and wherein the coverage metrics are variable and actively and automatically updated, and wherein the coverage metrics include an input robustness score and an output robustness score, wherein each of the input and output robustness scores define a percentage of coverage of a list of or of all possible output instructions of a particular type and name to variable mapping;
• wherein the input robustness score further accounts for:
• variations in input terminology;
• variations in input statement types;
• and
• variations in substring interpretation coverage;
• and wherein the output robustness score comprises:
• a percentage of coverage of a list of or of all possible output instructions and names to variable mapping;
• identifying and filling information gaps in extracted syntactical information, including:
• reading, from outputs of the control logic for extracting syntactical information, all terms and phrases relevant to coverage metrics and splitting the terms and phrases into key elements;
• identifying, with first looped control logic, variations for each of the key terms in a given context, and assigning a confidence score to each of the variations for each of the key terms;
• and
• identifying, with second looped control logic, input variations for each seed test input statement and user input statement, and assigning a confidence score to each of the seed test and user input statements;
• causing the human validators to manually evaluate outputs of each of the first and second looped control logics for identifying variations for each of the key terms and for each of the seed test input and user input statements;
• and
• synthesizing variations that actively adapt to fill identified information gaps;
• performing a coverage measurement of extracted syntactical information, including:
• applying the coverage metrics to the extracted syntactical information, including:
• measuring a seed expansion for variation in input terms;
• measuring a seed expansion for a variation in input statement types;
• measuring a seed expansion for substring interpretation coverage;
• measuring a seed expansion for output robustness;
• and
• aggregating and normalizing measured seed expansions for the variation in input terms, the variation in input statement types, the substring interpretation, and output robustness, and generating a coverage measurement score;
• and
• causing a human validator to evaluate the coverage measurement relative to predefined coverage metrics and selectively and continuously iterate to cause coverage measurement to increase LLM tool input robustness and output robustness from a first level to a second level greater than the first level, in both production and pre-production LLM tool processes, by causing the human validator to evaluate outputs of the second control logic by selectively and continuously adding to or eliminating examples from the supplementary documents;
• and
• by causing the human validator to convert the coverage measurement into an improved seed file and selectively and continuously iterating inputs to the control logic that synthesizes variations that actively adapt to fill identified information gaps to increase LLM tool input and output robustness from the first level to the second level greater than the first level in both production and pre-production LLM tool processes, wherein the method progressively reduces computational resource utilization, progressively increases computational efficiency, and progressively reduces reliance on the human validator.