Recognizing applications, screens, and user interface elements and recognizing user and/or agent interactions therewith using agentic artificial intelligence

One or more non-transitory computer-readable media storing one or more computer programs, the one or more computer programs configured to cause at least one processor to:

• access screenshots or video frames of displays associated with at least one of one or more user computing systems and one or more hosted environments representing the one or more user computing systems and access other information associated with the one or more user computing systems and/or the one or more hosted environments;
• and
• recognize the applications, screens, and user interface (UI) elements that are present in the screenshots or video frames using the screenshots or video frames and the other information, via an artificial intelligence (AI) agent that utilizes an AI model, wherein
• the AI model is configured to recognize the applications, screens, and UI elements in the screenshots or video frames without a priori knowledge,
• the AI agent recognizes the applications, screens, and UI elements by calling the AI model, and
• the AI agent receives output from the AI model and utilizes the output to infer one or more actions to perform.

One or more computing systems, comprising:

• memory storing computer program instructions;
• and
• at least one processor configured to execute the computer program instructions, wherein the computer program instructions are configured to cause the at least one processor to:
• access screenshots or video frames of displays associated with at least one of one or more user computing systems and one or more hosted environments representing the one or more user computing systems and access other information associated with the one or more computing systems, and
• recognize the applications, screens, and user interface (UI) elements that are present in the screenshots or video frames using the screenshots or video frames and the other information and recognize individual user interactions or AI agent interactions with the UI elements using the recognized applications, screens, and UI elements, via an artificial intelligence (AI) agent that utilizes an AI model, wherein
• the AI agent recognizes the applications, screens, UI elements, and user interactions or AI agent interactions by calling the AI model, and
• the AI agent receives output from the AI model and utilizes the output to infer one or more actions to perform.