# US 2025/0147647 A1 · User interface automation using robotic process automations that detect user interface elements not visible on a display and fills a form

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2025/0147647 A1
- Document ID: US20250147647A1
- Status: Published application
- Kind: A1
- Application number: 18/502,223
- Filed: 2023-11-06
- Published: 2025-05-08
- Earliest claimed: 2023-11-06
- Assignees: UiPath, Inc.
- Applicants: UiPath, Inc.
- Inventors: Cosmin VOICU; Ion MIRON; Dan MARCULET; Alexandru-Cristian PETRESCU
- Web page: https://patentgrep.com/patent/US20250147647A1

## CPC

- G06F 9/451: Execution arrangements for user interfaces
- G06F 40/174: Form filling; Merging
- G06F 40/106: Display of layout of documents; Previewing
- G06F 40/14: Tree-structured documents (parsing G06F40/205; validation G06F40/226)
- G06F 40/205: Parsing
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G06N 3/045: Combinations of networks
- G06N 3/08: Learning methods
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06N 20/00: Machine learning
- G06V 30/412: Layout analysis of documents structured with printed lines or input boxes, e.g. business forms or tables
- G06V 30/414: Extracting the geometrical structure, e.g. layout tree; Block segmentation, e.g. bounding boxes for graphics or text
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

A method is provided. The method is for detecting and automatically filling user interface (UI) elements of a page that are not visible on a screen. The method is executed by an interface engine implemented as a computer program within a computing environment. The method includes analyzing document object model (DOM) of the page to extract DOM text of relevant UI elements of the UI elements. The method includes performing computer vision (CV) analysis to determine types the relevant UI elements and extrapolating target UI elements of an invisible region of the page from the relevant UI elements.

## Claims

### Claim 1

A method for detecting and automatically filling one or more user interface (UI) elements of a page that are not visible on a screen, the method being executed by an interface engine implemented as a computer program within a computing environment, the method comprising:
analyzing, by the interface engine, document object model (DOM) of the page to extract DOM text of one or more relevant UI elements of the one or more UI elements;
performing, by the interface engine, computer vision (CV) analysis to determine one or more types the one or more relevant UI elements; and
extrapolating, by the interface engine, one or more target UI elements of an invisible region of the page from the one or more relevant UI elements.

### Claim 2 (depends on 1)

The method of claim 1, wherein the method includes linking the DOM text to the one or more target UI elements based on the one or more types and the one or more relevant UI elements as determined by the CV analysis.

### Claim 3 (depends on 1)

The method of claim 1, wherein the extrapolating of the one or more target UI elements by the interface engine comprises searching for similar hypertext markup language (HTML) structures from the DOM.

### Claim 4 (depends on 1)

The method of claim 1, wherein the one or more UI elements comprises one or more input fields of the page.

### Claim 5 (depends on 1)

The method of claim 1, wherein the interface engine extracts data from a source to paste into at least the one or more target UI elements.

### Claim 6 (depends on 1)

The method of claim 1, wherein the one or more types of the one or more relevant UI elements comprise DOM types, framework types, CV types, or element definition types.

### Claim 7 (depends on 1)

The method of claim 1, wherein the performing of the CV by the interface engine comprises capturing a visible region of the page to identify the one or more types of the one or more relevant UI elements in the visible region.

### Claim 8 (depends on 1)

The method of claim 1, wherein the interface engine inputs data into the one or more target UI elements without scrolling the page.

### Claim 9 (depends on 1)

The method of claim 1, wherein the page comprises a long form.

### Claim 10 (depends on 1)

The method of claim 1, wherein the invisible region of the page comprises a destination form of a subsequent.

### Claim 11

A system comprising:
a memory storing code of an interface engine for detecting and automatically filling one or more user interface (UI) elements of a page that are not visible on a screen; and
at least one processor configured to execute the code to cause within the system:
analyzing, by the interface engine, document object model (DOM) of the page to extract DOM text of one or more relevant UI elements of the one or more UI elements;
performing, by the interface engine, computer vision (CV) analysis to determine one or more types the one or more relevant UI elements; and
extrapolating, by the interface engine, one or more target UI elements of an invisible region of the page from the one or more relevant UI elements.

### Claim 12 (depends on 11)

The system of claim 11, wherein the interface engine links the DOM text to the one or more target UI elements based on the one or more types and the one or more relevant UI elements as determined by the CV analysis.

### Claim 13 (depends on 11)

The system of claim 11, wherein the extrapolating of the one or more target UI elements by the interface engine comprises searching for similar hypertext markup language (HTML) structures from the DOM.

### Claim 14 (depends on 11)

The system of claim 11, wherein the one or more UI elements comprises one or more input fields of the page.

### Claim 15 (depends on 11)

The system of claim 11, wherein the interface engine extracts data from a source to paste into at least the one or more target UI elements.

### Claim 16 (depends on 11)

The system of claim 11, wherein the one or more types of the one or more relevant UI elements comprise DOM types, framework types, CV types, or element definition types.

### Claim 17 (depends on 11)

The system of claim 11, wherein the performing of the CV by the interface engine comprises capturing a visible region of the page to identify the one or more types of the one or more relevant UI elements in the visible region.

### Claim 18 (depends on 11)

The system of claim 11, wherein the interface engine inputs data into the one or more target UI elements without scrolling the page.

### Claim 19 (depends on 11)

The system of claim 11, wherein the page comprises a long form.

### Claim 20 (depends on 11)

The system of claim 11, wherein the invisible region of the page comprises a destination form of a subsequent.
