Interactive interface task automation utilizing generative artificial intelligence (ai) action models improved with retrieval-augmented generation (rag)

A system comprising:

• a processing system having a processor;
• and
• a computer memory including instructions that, when executed by the processing system, cause the system to carry out operations comprising:
• in response to receiving user input indicating an actionable task to be automatically performed on an interactive interface, obtaining prior session information corresponding to the actionable task and the interactive interface;
• providing a session plan generation prompt, which includes the prior session information, to a generative AI action model to generate a session plan that includes a set of actions for performing the actionable task;
• identifying an interactive element heatmap from a RAG database and visual context information from a visual-based generative AI model for a first action from the set of actions, wherein the interactive element heatmap indicates interactive elements usage by previous users;
• providing an action execution prompt, which includes the session plan, the interactive element heatmap for the first action, and the visual context information for the first action, to the generative AI action model to generate an executable first action scheme for accomplishing the first action;
• and
• performing the actionable task based on performing the executable first action scheme.

A computer-implemented method for performing one or more tasks based on one or more generative artificial intelligence (AI) models using retrieval-augmented generation (RAG) inputs, comprising:

• in response to receiving user input indicating an actionable task to be automatically performed on an interactive interface, obtaining prior session information corresponding to the actionable task and the interactive interface;
• providing a session plan generation prompt, which includes the prior session information, to a generative AI action model to generate a session plan that includes a set of actions for performing the actionable task;
• providing an action execution prompt, which includes the session plan, an interactive element heatmap for a first action of the set of actions, and visual context information for the first action, to the generative AI action model to generate an executable first action scheme for accomplishing the first action;
• performing the executable first action scheme;
• and
• performing the actionable task based on performing each action in the session plan.