User interface action tracking for quality evaluation of AI-generated content

A system comprising:

• a processing device including a processing system;
• an artificial intelligence (AI) content evaluation tool stored in memory and executable by the processing system to:
• receive, from a generative artificial intelligence (AI) model, a set of AI-generated instructions that identify steps for performing a task within an application;
• select, from an interaction index that defines a plurality of user interface interactions, checkpoint interactions that each satisfy a similarity metric with a corresponding step in the set of AI-generated instructions;
• detect interactions between a user and a user interface of the application over an observation period;
• identify a subset of the checkpoint interactions completed by the user within the observation period;
• and evaluate a metric that quantifies user success with respect to performing the task, the metric depending at least in part on the subset of the checkpoint interactions completed by the user within the observation period;
• and transmit a quality alert in response to determining that the metric satisfies low-quality criteria.

One or more tangible computer-readable storage media encoding processor-executable instructions for performing a computer process, the computer process comprising:

• receiving, from a generative artificial intelligence (AI) model, a set of AI-generated instructions that identifies steps for performing a task within an application;
• selecting, from an interaction index that defines a plurality of interactions with a user interface of an application, checkpoint interactions that each satisfy a similarity metric with a corresponding step in the set of AI-generated instructions;
• determining, based on detected user interactions with a user interface of the application, a subset of the checkpoint interactions completed by a user within an observation period;
• and
• evaluating a metric that quantifies user success with respect to performing the task, the metric depending at least in part on the subset of the checkpoint interactions completed by the user within the observation period;
• and
• based on the metric satisfying predefined criteria, generate and transmit a quality alert that includes the set of AI-generated instructions and an input query that was used, by the generative AI model, as a basis for generating the set of AI-generated instructions.