# US 2021/0342622 A1 · Text detection, caret tracking, and active element detection

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0342622 A1
- Document ID: US20210342622A1
- Status: Published application
- Kind: A1
- Application number: 16/865,832
- Filed: 2020-05-04
- Published: 2021-11-04
- Earliest claimed: 2020-05-01
- Assignees: UiPath, Inc.
- Applicants: UiPath, Inc.
- Inventors: Vaclav SKARDA
- Web page: https://patentgrep.com/patent/US20210342622A1

## CPC

- G06F 3/0481: based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
- G06V 10/25: Determination of region of interest [ROI] or a volume of interest [VOI]
- G06V 10/40: Extraction of image or video features
- G06V 20/41: {Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items (segmenting video sequences G06V20/49)}
- G06V 30/19013: {Comparing pixel values or logical combinations thereof, or feature values having positional relevance, e.g. template matching (specially adapted for image segmentation G06T7/10; specially adapted for the analysis of motion G06T7/20; specially adapted for image alignment G06T7/30; specially adapted for the calculation of depth from stereo images G06T7/50; specially adapted for position determination G06T7/70)}
- G06V 30/10: Character recognition
- G06V 2201/02: Recognising information on displays, dials, clocks
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

Detection of typed and/or pasted text, caret tracking, and active element detection for a computing system are disclosed. The location on the screen associated with a computing system where the user has been typing or pasting text, potentially including hot keys or other keys that do not cause visible characters to appear, can be identified and the physical position on the screen where typing or pasting occurred can be provided based on the current resolution of where one or more characters appeared, where the cursor was blinking, or both. This can be done by identifying locations on the screen where changes occurred and performing text recognition and/or caret detection on these locations. The physical position of the typing or pasting activity allows determination of an active or focused element in an application displayed on the screen.

## Claims

### Claim 1

A computer program comprising caret tracking logic embodied on a non-transitory computer-readable medium, the program configured to cause at least one processor to:
determine one or more regions where a current screenshot frame differs from a previous screenshot frame, the current screenshot frame and the previous screenshot frame comprising an entire screen of a display or a portion thereof;
calculate pixel changes for the one or more regions;
project the calculated pixel changes to a binary matrix for each of the one or more regions;
for each binary matrix, extract blinking caret region candidates and join members in the binary matrix; and
extract shapes from the members and validate the shapes of the members against a shape of the caret.

### Claim 2 (depends on 1)

The computer program of claim 1, wherein the members are connected shapes present in the binary matrix.

### Claim 3 (depends on 1)

The computer program of claim 1, wherein
the joining of the members in each binary matrix is performed using a Connected Components algorithm, and
each member is a set of pixels where each pixel is connected to all other pixels in the member.

### Claim 4 (depends on 1)

The computer program of claim 1, wherein a shape of the caret comprises a rectangle.

### Claim 5 (depends on 1)

The computer program of claim 1, wherein when a shape of a member is validated, the program is further configured to cause the at least one processor to:
store a position and size of the validated member and a time stamp associated with the current screenshot frame as validated member shape data; and
compare the validated member shape data to a confirmation queue.

### Claim 6 (depends on 5)

The computer program of claim 5, wherein the comparison comprises determining whether the validated member blinks at a frequency within a tolerance.

### Claim 7 (depends on 6)

The computer program of claim 6, wherein the program is further configured to cause the at least one processor to:
determine that the caret has moved to a new location in the current screenshot frame based on the member shape data of the caret and the frequency at which the caret blinks.

### Claim 8 (depends on 6)

The computer program of claim 6, wherein when the validated member blinks at the frequency within the tolerance, the program is further configured to cause the at least one processor to:
save the validated member as the caret;
determine a graphical element that includes coordinates of the caret; and
set the graphical element as an active element.

### Claim 9 (depends on 1)

The computer program of claim 1, wherein the binary matrix represents pixels with a change between the current screenshot frame and the previous screenshot frame with a “1” and pixels where no change occurred with a “0”.

### Claim 10 (depends on 1)

The computer program of claim 1, wherein
a predetermined number of regions are supported, and
when the number of the one or more regions exceeds the number of supported regions, the current screenshot frame is ignored and the caret tracking logic proceeds to a next screenshot frame.

### Claim 11 (depends on 1)

The computer program of claim 1, wherein when more than a predetermined percentage of the pixels of the current screenshot frame differ from the previous screenshot frame, the current screenshot frame is ignored and the caret tracking logic proceeds to a next screenshot frame.

### Claim 12 (depends on 1)

The computer program of claim 1, wherein the program is further configured to cause the at least one processor to:
store coordinates of a location of a most recent mouse click on a screen;
identify a region near the mouse click, a graphical element including the coordinates of the last mouse click, or both, as a region where the caret is likely to appear; and
focus the caret tracking logic on the region or the graphical element including the coordinates.

### Claim 13 (depends on 12)

The computer program of claim 12, the program is further configured to cause the at least one processor to:
detect that a user pressed a tab key; and
refocus the caret tracking logic on a next graphical element in the current screenshot frame responsive to detecting that the user pressed the tab key.

### Claim 14 (depends on 12)

The computer program of claim 12, the program is further configured to cause the at least one processor to:
detect that a user pressed a tab key; and
refocus the caret tracking logic on an entire current screenshot frame responsive to detecting that the user pressed the tab key.

### Claim 15 (depends on 1)

The computer program of claim 1, wherein the program is a robotic process automation (RPA) robot.

### Claim 16

A computer program embodied on a non-transitory computer-readable medium, the program configured to cause at least one processor to:
analyze one or more regions where a current screenshot frame differs from a previous screenshot frame, the current screenshot frame and the previous screenshot frame comprising an entire screen of a display or a portion thereof;
identify one or more caret candidates based on the analysis;
perform validation to identify that a caret candidate of the one or more caret candidates is blinking;
identify a location of a caret based on the validation; and
set a graphical element in the current screenshot frame with the location of the caret as an active element.

### Claim 17 (depends on 16)

The computer program of claim 16, wherein the analysis of the one or more regions where the current screenshot frame differs from the previous screenshot frame comprises:
calculating pixel changes for the one or more regions;
projecting the calculated pixel changes to a binary matrix for each of the one or more regions; and
for each binary matrix, extracting blinking caret region candidates and joining members in the binary matrix, wherein
the members are connected shapes present in the binary matrix.

### Claim 18 (depends on 16)

The computer program of claim 16, wherein when a shape of a member is validated, the program is further configured to cause the at least one processor to:
store a position and size of the validated member and a time stamp associated with the current screenshot frame as validated member shape data;
compare the validated member shape data to a confirmation queue by determining whether the validated member blinks at a frequency within a tolerance;
determine that the caret has moved to a new location in the current screenshot frame based on the member shape data of the caret and the frequency at which the caret blinks;
save the validated member as the caret;
determine a graphical element that includes coordinates of the caret; and
set the graphical element as an active element.

### Claim 19 (depends on 16)

The computer program of claim 16, wherein
a predetermined number of regions are supported, and
when the number of the one or more regions exceeds the number of supported regions, the current screenshot frame is ignored and the caret tracking logic proceeds to a next screenshot frame.

### Claim 20

A computer-implemented method for performing caret tracking for robotic process automation (RPA), comprising:
analyzing one or more regions where a current screenshot frame differs from a previous screenshot frame, by an RPA robot, the current screenshot frame and the previous screenshot frame comprising an entire screen of a display or a portion thereof;
identifying one or more caret candidates based on the analysis, by the RPA robot;
performing validation to identify that a caret candidate of the one or more caret candidates is blinking, by the RPA robot;
identifying a location of a caret based on the validation, by the RPA robot; and
setting a graphical element in the current screenshot frame with the location of the caret as an active element, by the RPA robot.
