# US 2026/0064558 A1 · Recognizing applications, screens, and user interface elements and recognizing user and/or agent interactions therewith using agentic artificial intelligence

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2026/0064558 A1
- Document ID: US20260064558A1
- Status: Published application
- Kind: A1
- Application number: 19/385,351
- Filed: 2025-11-11
- Published: 2026-03-05
- Earliest claimed: 2020-10-14
- Assignees: UiPath, Inc.
- Applicants: UiPath, Inc.
- Inventors: Michael Aristo LEONARD, II; Prabhdeep SINGH
- Web page: https://patentgrep.com/patent/US20260064558A1

## CPC

- G06F 11/3438: {monitoring of user actions (tracking the activity of the user H04L67/535)}
- G06V 10/774: Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
- G06V 10/82: using neural networks
- G06V 30/153: {using recognition of characters or words}
- G06F 9/451: Execution arrangements for user interfaces
- G06F 9/45529: {Embedded in an application, e.g. JavaScript in a Web browser}
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G06N 3/0455: Auto-encoder networks; Encoder-decoder networks
- G06N 3/0475: Generative networks
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06V 2201/02: Recognising information on displays, dials, clocks
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

Techniques for recognizing applications, screens, and UI elements and for recognizing user interactions and/or AI agent interactions with the applications, screens, and UI elements using agentic artificial intelligence (AI) are disclosed. An AI model may facilitate such recognition without other system inputs, such as system-level information (e.g., key presses, mouse clicks, locations, operating system operations, etc.) or application-level information (e.g., information from an application programming interface (API) from a software application executing on a computing system).

## Claims

### Claim 1

A system, comprising:
one or more user computing systems comprising respective recorder processes, the recorder processes comprising a process on the one or more user computing systems or one or more hosted environments representing the one or more user computing systems; and
a server configured to recognize applications, screens, and user interface (UI) elements and to recognize user interactions and/or AI agent interactions with the applications, screens, and UI elements using an agentic artificial intelligence (AI) agent that utilizes an AI model, wherein
the respective recorder processes are configured to send one or more screenshots or video frames, and other information, to storage comprising one or more ephemeral or persistent repositories accessible by the server,
the server is configured to recognize the applications, screens, and UI elements that are present in the recorded screenshots or video frames using the recorded screenshots or video frames and the other information and recognize individual user interactions or AI agent interactions with the UI elements using the recognized applications, screens, and UI elements, via the AI agent,
the AI agent recognizes the applications, screens, UI elements, and user or AI agent interactions by calling the AI model, and
the AI agent receives output from the AI model and utilizes the output to infer one or more actions to perform.

### Claim 2 (depends on 1)

The system of claim 1, wherein the one or more screenshots or video frames comprise a live on-screen state and the other information comprises at least one of a session history, an action log, and one or more control events.

### Claim 3 (depends on 1)

The system of claim 1, wherein the recorder processes are server-side and/or containerized processes that capture a physical or virtual display.

### Claim 4 (depends on 1)

The system of claim 1, wherein the AI model is part of or comprises a computer vision (CV) pipeline.

### Claim 5 (depends on 1)

The system of claim 1, wherein the individual user interactions or AI agent interactions comprise at least one of button presses, entry of single characters or character sequences, selection of active UI elements, menu selections, screen changes, voice inputs, gestures, providing biometric information, and haptic interactions.

### Claim 6 (depends on 1)

The system of claim 1, wherein
the AI model is trained to recognize the individual user interactions or AI agent interactions with the UI elements, and
the training comprises comparing two or more consecutive screenshots or video frames and determining that a typed character appeared from one screenshot to another, a button was pressed, or a menu selection occurred.

### Claim 7 (depends on 1)

The system of claim 1, wherein the other information comprises at least one of a web browser history, one or more heat maps, key presses, mouse clicks, locations of mouse clicks and/or graphical elements on the display that a user is interacting with, locations where the user was looking on the display, time stamps associated with the screenshots or video frames, text that the user entered, content that the user scrolled past, a time that the user stopped on a part of content shown in the display, what application the user is interacting with, voice inputs, gestures, emotion information, biometrics, information pertaining to periods of no user activity, haptic information, and multi-touch input information.

### Claim 8 (depends on 1)

The system of claim 1, wherein the respective recorder processes are implemented as feedback loop processes that continuously or periodically compare a current screenshot or video frame to a previous screenshot or video frame and identify one or more locations where changes between the current screenshot or video frame and the previous screenshot or video frame occurred.

### Claim 9 (depends on 8)

The system of claim 8, wherein the respective recorder processes are further configured to:
perform optical character recognition (OCR) on the one or more locations where the changes occurred;
compare results of the OCR to content of a keyboard queue to determine whether a match exists; and
when a match exists, link text associated with the match to a respective location.

### Claim 10 (depends on 1)

The system of claim 1, wherein the AI model is configured to recognize the applications, screens, and UI elements in the screenshots or video frames without a priori knowledge.

### Claim 11

One or more non-transitory computer-readable media storing one or more computer programs, the one or more computer programs configured to cause at least one processor to:
access screenshots or video frames of displays associated with at least one of one or more user computing systems and one or more hosted environments representing the one or more user computing systems and access other information associated with the one or more user computing systems and/or the one or more hosted environments; and
recognize the applications, screens, and user interface (UI) elements that are present in the screenshots or video frames using the screenshots or video frames and the other information, via an artificial intelligence (AI) agent that utilizes an AI model, wherein
the AI model is configured to recognize the applications, screens, and UI elements in the screenshots or video frames without a priori knowledge,
the AI agent recognizes the applications, screens, and UI elements by calling the AI model, and
the AI agent receives output from the AI model and utilizes the output to infer one or more actions to perform.

### Claim 12 (depends on 11)

The one or more non-transitory computer-readable media of claim 11, wherein the one or more screenshots or video frames comprise a live on-screen state and the other information comprises at least one of a session history, an action log, and one or more control events.

### Claim 13 (depends on 11)

The one or more non-transitory computer-readable media of claim 11, wherein the AI model is part of or comprises a computer vision (CV) pipeline.

### Claim 14 (depends on 11)

The one or more non-transitory computer-readable media of claim 11, wherein the other information comprises at least one of a web browser history, one or more heat maps, key presses, mouse clicks, locations of mouse clicks and/or graphical elements on the display that a user is interacting with, locations where the user was looking on the display, time stamps associated with the screenshots or video frames, text that the user entered, content that the user scrolled past, a time that the user stopped on a part of content shown in the display, what application the user is interacting with, voice inputs, gestures, emotion information, biometrics, information pertaining to periods of no user activity, haptic information, and multi-touch input information.

### Claim 15

One or more computing systems, comprising:
memory storing computer program instructions; and
at least one processor configured to execute the computer program instructions, wherein the computer program instructions are configured to cause the at least one processor to:
access screenshots or video frames of displays associated with at least one of one or more user computing systems and one or more hosted environments representing the one or more user computing systems and access other information associated with the one or more computing systems, and
recognize the applications, screens, and user interface (UI) elements that are present in the screenshots or video frames using the screenshots or video frames and the other information and recognize individual user interactions or AI agent interactions with the UI elements using the recognized applications, screens, and UI elements, via an artificial intelligence (AI) agent that utilizes an AI model, wherein
the AI agent recognizes the applications, screens, UI elements, and user interactions or AI agent interactions by calling the AI model, and
the AI agent receives output from the AI model and utilizes the output to infer one or more actions to perform.

### Claim 16 (depends on 15)

The one or more computing systems of claim 15, wherein the AI model is configured to recognize the applications, screens, and UI elements in the screenshots or video frames without a priori knowledge.

### Claim 17 (depends on 15)

The one or more computing systems of claim 15, wherein the one or more screenshots or video frames comprise a live on-screen state and the other information comprises at least one of a session history, an action log, and one or more control events.

### Claim 18 (depends on 15)

The one or more computing systems of claim 15, wherein the AI model is part of or comprises a computer vision (CV) pipeline.

### Claim 19 (depends on 15)

The one or more computing systems of claim 15, wherein the individual user interactions or AI agent interactions comprise at least one of button presses, entry of single characters or character sequences, selection of active UI elements, menu selections, screen changes, voice inputs, gestures, providing biometric information, and haptic interactions.

### Claim 20 (depends on 15)

The one or more computing systems of claim 15, wherein the other information comprises at least one of a web browser history, one or more heat maps, key presses, mouse clicks, locations of mouse clicks and/or graphical elements on the display that a user is interacting with, locations where the user was looking on the display, time stamps associated with the screenshots or video frames, text that the user entered, content that the user scrolled past, a time that the user stopped on a part of content shown in the display, what application the user is interacting with, voice inputs, gestures, emotion information, biometrics, information pertaining to periods of no user activity, haptic information, and multi-touch input information.
