US 11,080,548 B1 · Text detection, caret tracking, and active element detection

Abstract

Detection of typed and/or pasted text, caret tracking, and active element detection for a computing system are disclosed. The location on the screen associated with a computing system where the user has been typing or pasting text, potentially including hot keys or other keys that do not cause visible characters to appear, can be identified and the physical position on the screen where typing or pasting occurred can be provided based on the current resolution of where one or more characters appeared, where the cursor was blinking, or both. This can be done by identifying locations on the screen where changes occurred and performing text recognition and/or caret detection on these locations. The physical position of the typing or pasting activity allows determination of an active or focused element in an application displayed on the screen.

Claims40

Dependent claims

A computer program comprising typed text detection logic embodied on a non-transitory computer-readable medium, the program configured to cause at least one processor to:

generate a keyboard character queue comprising key press events that occurred during a time window;
determine one or more regions where a current screenshot frame differs from a previous screenshot frame, the current screenshot frame and the previous screenshot frame comprising an entire screen of a display or a portion thereof;
extract one or more connected members in the one or more determined regions and perform optical character recognition (OCR) on the extracted one or more connected members, producing one or more recognized characters, one or more recognized character sequences, or both, and respective positions;
compare the one or more recognized characters, the one or more recognized character sequences, or both, to the keyboard character queue;
and
when a character or character sequence matches a character or character sequence of the keyboard character queue, remove the character or character sequence from the keyboard character queue.

3 more elements

A computer-implemented method for typed text detection for robotic process automation (RPA), comprising:

generating a keyboard character queue comprising key press events that occurred during a time window, by an RPA robot, the key press events comprising a character of a key that was pressed and a time that the key was pressed;
determining, by the RPA robot, one or more regions where a current screenshot frame differs from a previous screenshot frame, the current screenshot frame and the previous screenshot frame comprising an entire screen of a display or a portion thereof;
extracting one or more connected members in the one or more determined regions and performing optical character recognition (OCR) on the extracted one or more connected members, by the RPA robot, producing one or more recognized characters, one or more recognized character sequences, or both, and respective positions;
comparing the one or more recognized characters, the one or more recognized character sequences, or both, to the keyboard character queue, by the RPA robot;
and
when a character or character sequence matches a character or character sequence of the keyboard character queue:
removing the character or character sequence from the keyboard character queue, by the RPA robot, and
setting a graphical element in the current screenshot frame with a position including the position of the matching character or character sequence as an active element, by the RPA robot.

5 more elements

Dependencies are inferred from the claim text.

Text detection, caret tracking, and active element detection

Life of this patent

Text detection, caret tracking, and active element detection

Life of this patent

Abstract

Claims40