# US 2021/0342623 A1 · Text detection, caret tracking, and active element detection

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0342623 A1
- Document ID: US20210342623A1
- Status: Published application
- Kind: A1
- Application number: 16/865,978
- Filed: 2020-05-04
- Published: 2021-11-04
- Earliest claimed: 2020-05-01
- Assignees: UiPath, Inc.
- Applicants: UiPath, Inc.
- Inventors: Vaclav SKARDA
- Web page: https://patentgrep.com/patent/US20210342623A1

## CPC

- G06F 3/0481: based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
- G06F 3/0233: {Character input methods}
- G06V 10/25: Determination of region of interest [ROI] or a volume of interest [VOI]
- G06V 10/40: Extraction of image or video features
- G06V 10/44: Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
- G06V 10/74: Image or video pattern matching; Proximity measures in feature spaces
- G06V 20/41: {Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items (segmenting video sequences G06V20/49)}
- G06V 20/46: {Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames}
- G06V 30/19013: {Comparing pixel values or logical combinations thereof, or feature values having positional relevance, e.g. template matching (specially adapted for image segmentation G06T7/10; specially adapted for the analysis of motion G06T7/20; specially adapted for image alignment G06T7/30; specially adapted for the calculation of depth from stereo images G06T7/50; specially adapted for position determination G06T7/70)}
- G06V 30/41: Analysis of document content (recognition of printed characters based on code marks G06V30/224)
- G06F 11/3438: {monitoring of user actions (tracking the activity of the user H04L67/535)}
- G06V 30/10: Character recognition
- G06V 2201/02: Recognising information on displays, dials, clocks
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

Detection of typed and/or pasted text, caret tracking, and active element detection for a computing system are disclosed. The location on the screen associated with a computing system where the user has been typing or pasting text, potentially including hot keys or other keys that do not cause visible characters to appear, can be identified and the physical position on the screen where typing or pasting occurred can be provided based on the current resolution of where one or more characters appeared, where the cursor was blinking, or both. This can be done by identifying locations on the screen where changes occurred and performing text recognition and/or caret detection on these locations. The physical position of the typing or pasting activity allows determination of an active or focused element in an application displayed on the screen.

## Claims

### Claim 1

A computer program for preprocessing screenshot frames and detecting active elements embodied on a non-transitory computer-readable medium, the program configured to cause at least one processor to:
run computer vision (CV) preprocessing on a previous screenshot frame;
determine types and locations of graphical elements in the previous screenshot frame;
determine that a change occurred in a region where a current screenshot frame differs from the previous screenshot frame, the region at least partially included within an area of one or more graphical elements that permit text entry; and
when the change occurred in one graphical element that permits text entry, set the graphical element where the change occurred as an active element.

### Claim 2 (depends on 1)

The computer program of claim 1, wherein the program is further configured to cause the at least one processor to:
generate a keyboard character queue of key press events that occurred during a time window;
determine that changes occurred in two or more graphical elements that permit text entry between the current screenshot frame and the previous screenshot frame;
determine newly added key press events that occurred between the previous screenshot frame and the current screenshot frame;
perform optical character recognition (OCR) on regions of the two or more graphical elements where the changes occurred;
match the newly added key press events to one of the two or more graphical elements where the changes occurred; and
set the matched graphical element as the active element.

### Claim 3 (depends on 1)

The computer program of claim 1, wherein the program is further configured to cause the at least one processor to:
fetch a next screenshot frame; and
repeat the process using the next screenshot frame as the current screenshot frame and using the current screenshot frame as the previous screenshot frame.

### Claim 4 (depends on 1)

The computer program of claim 1, wherein the program is further configured to cause the at least one processor to:
pause determination of changes between screenshot frames while the CV preprocessing is running.

### Claim 5 (depends on 1)

The computer program of claim 1, wherein when the current screenshot frame changes more than a threshold relative to the previous screenshot frame, the program is further configured to cause the at least one processor to:
wait until a new current screenshot frame does not change more than the threshold relative to a new previous screenshot frame; and
when the new current screenshot frame does not change more than the threshold relative to the new previous screenshot frame:
run the CV preprocessing on the new current screenshot frame, and determine types and locations of graphical elements in the new current screenshot frame.

### Claim 6 (depends on 1)

The computer program of claim 1, wherein the program is further configured to cause the at least one processor to:
perform caret tracking to identify a caret; and
identify the active element in the current screenshot frame as the graphical element that includes the caret.

### Claim 7 (depends on 1)

The computer program of claim 1, wherein
the current screenshot frame and previous screenshot frame are compressed in a lossy format, and
the determining of the region where the current screenshot frame differs from the previous screenshot frame comprises applying one or more tolerances to the region.

### Claim 8 (depends on 1)

The computer program of claim 1, wherein when determining the region where the current screenshot frame differs from the previous screenshot frame, the program is further configured to cause the at least one processor to:
apply fuzzy image matching to a region from the current screenshot frame and a corresponding region from the previous screenshot frame to determine whether a change occurred in the region.

### Claim 9 (depends on 1)

The computer program of claim 1, wherein the program is a robotic process automation (RPA) robot.

### Claim 10 (depends on 1)

The computer program of claim 1, wherein when more than a predetermined percentage of the pixels of the current screenshot frame differ from the previous screenshot frame, the current screenshot frame is ignored and the logic proceeds to a next screenshot frame.

### Claim 11

A computer program for preprocessing screenshot frames and detecting active elements embodied on a non-transitory computer-readable medium, the program configured to cause at least one processor to:
run computer vision (CV) preprocessing on a previous screenshot frame;
determine types and locations of graphical elements in the previous screenshot frame;
determine that a change occurred in a region where a current screenshot frame differs from the previous screenshot frame, the region at least partially included within an area of one or more graphical elements that permit text entry;
when the change occurred in one graphical element that permits text entry, set the graphical element where the change occurred as an active element; and
when the change occurred in more than one graphical element that permits text entry:
generate a keyboard character queue of key press events that occurred during a time window,
determine newly added key press events that occurred between the previous screenshot frame and the current screenshot frame,
perform optical character recognition (OCR) on regions of the two or more graphical elements where the changes occurred,
match the newly added key press events to one of the two or more graphical elements where the changes occurred, and
set the matched graphical element as the active element.

### Claim 12 (depends on 11)

The computer program of claim 11, wherein the program is further configured to cause the at least one processor to:
fetch a next screenshot frame; and
repeat the process using the next screenshot frame as the current screenshot frame and using the current screenshot frame as the previous screenshot frame.

### Claim 13 (depends on 11)

The computer program of claim 11, wherein the program is further configured to cause the at least one processor to:
pause determination of changes between screenshot frames while the CV preprocessing is running.

### Claim 14 (depends on 11)

The computer program of claim 11, wherein when the current screenshot frame changes more than a threshold relative to the previous screenshot frame, the program is further configured to cause the at least one processor to:
wait until a new current screenshot frame does not change more than the threshold relative to a new previous screenshot frame; and
when the new current screenshot frame does not change more than the threshold relative to the new previous screenshot frame:
run the CV preprocessing on the new current screenshot frame, and
determine types and locations of graphical elements in the new current screenshot frame.

### Claim 15 (depends on 11)

The computer program of claim 11, wherein the program is further configured to cause the at least one processor to:
perform caret tracking to identify a caret; and
identify an active element in the current screenshot frame that includes the caret.

### Claim 16 (depends on 11)

The computer program of claim 11, wherein the program is a robotic process automation (RPA) robot.

### Claim 17 (depends on 11)

The computer program of claim 11, wherein when more than a predetermined percentage of the pixels of the current screenshot frame differ from the previous screenshot frame, the current screenshot frame is ignored and the logic proceeds to a next screenshot frame.

### Claim 18

A computer-implemented method for preprocessing screenshot frames and detecting active elements for robotic process automation (RPA), comprising:
running computer vision (CV) preprocessing on a previous screenshot frame, by an RPA robot;
determining types and locations of graphical elements in the previous screenshot frame;
determining that a change occurred in a region where a current screenshot frame differs from the previous screenshot frame, by the RPA robot, the region at least partially included within an area of one or more graphical elements that permit text entry; and
when the change occurred in one graphical element that permits text entry, setting the graphical element where the change occurred as an active element, by the RPA robot.

### Claim 19 (depends on 18)

The computer-implemented method of claim 18, further comprising:
generating a keyboard character queue of key press events that occurred during a time window, by the RPA robot;
determining that changes occurred in two or more graphical elements that permit text entry between the current screenshot frame and the previous screenshot frame, by the RPA robot;
determining newly added key press events that occurred between the previous screenshot frame and the current screenshot frame, by the RPA robot;
performing optical character recognition (OCR) on regions of the two or more graphical elements where the changes occurred, by the RPA robot;
matching the newly added key press events to one of the two or more graphical elements where the changes occurred, by the RPA robot; and
setting the matched graphical element as the active element, by the RPA robot.

### Claim 20 (depends on 18)

The computer-implemented method of claim 18, wherein when the current screenshot frame changes more than a threshold relative to the previous screenshot frame, the method further comprises:
waiting until a new current screenshot frame does not change more than the threshold relative to a new previous screenshot frame, by the RPA robot; and
when the new current screenshot frame does not change more than the threshold relative to the new previous screenshot frame:
running the CV preprocessing on the new current screenshot frame, by the RPA robot, and
determining types and locations of graphical elements in the new current screenshot frame, by the RPA robot.

### Claim 21 (depends on 18)

The computer-implemented method of claim 18, further comprising:
performing caret tracking to identify a caret, by the RPA robot; and
identifying the active element in the current screenshot frame as the graphical element that includes the caret, by the RPA robot.
