US 12,596,343 B2 · Graphical element search technique selection, fuzzy logic selection of anchors and targets, and/or hierarchical graphical element identification for robotic process automation

Abstract

Graphical element search technique selection, fuzzy logic selection for anchors and targets, and hierarchical graphical element identification for robotic process automation (RPA) are disclosed. The fuzzy logic selection of anchors and targets may be part of a larger, tiered, or hierarchical process for identifying graphical elements in the UI. When a selector for a UI element is not found with at least a confidence threshold, similar elements potentially corresponding to the selector for a UI element target may be searched based on fuzzy matching of the target and corresponding anchor(s). Geometric matching may also be employed between the target UI element and its respective anchor(s). The combination of fuzzy matching and geometric matching may allow for more flexible and accurate identification of the exact selector with which an RPA robot is attempting to interact.

Claims29

Dependent claims

A computer-implemented method, comprising:

designating a target user interface (UI) element in an image of a portion or all of a Ul of an application or of a UI of a live application on which automation is to be performed at design time, by a robotic process automation (RPA) designer application, wherein the designating comprises a user selecting the UI element directly on the Ul of the application or the Ul of the live application displayed on a display device;
receiving a selection of one or more search algorithms of a plurality of search algorithms at design time, by the RPA designer application via a selection options pane, wherein the one or more search algorithms comprise at least one fuzzy selector;
and
configuring an activity of an RPA workflow for an RPA robot to execute the selected one or more search algorithms comprising the at least one fuzzy selector at design time, by the RPA designer application, wherein
an accuracy of the at least one fuzzy selector is configurable via the selection options pane.

2 more elements

A non-transitory computer-readable medium storing a computer program for a robotic process automation (RPA) designer application, the computer program configured to cause at least one processor to:

designate a target user interface (UI) element in an image of a portion or all of a UI of an application or of a UI of a live application on which automation is to be performed at design time, wherein the designating comprises a user selecting the UI element directly on the UI of the application or the Ul of the live application displayed on a display device;
receive a selection of one or more search algorithms of a plurality of search algorithms at design time via a selection options pane, wherein the one or more search algorithms comprise at least one fuzzy selector;
and
configure an activity of an RPA workflow te for an RPA robot to execute the selected one or more search algorithms at design time, wherein
the one or more search algorithms comprise a selector search, a fuzzy selector search performed via the at least one fuzzy selector, a target and anchor search, a fuzzy target and anchor search, image matching, text matching, computer vision (CV) matching, or a combination thereof, and
an accuracy of the at least one fuzzy selector is configurable via the selection options pane.

3 more elements

A computing system, comprising:

memory storing computer program instructions;
and
at least one processor configured to execute the computer program instructions, wherein the computer program instructions are configured to cause at least one processor to:
designate a target user interface (UI) element in an image of a portion or all of a UI of an application or of a UI of a live application on which automation is to be performed at design time, wherein the designating comprises a user selecting the UI element directly on the UI of the application or the UI of the live application displayed on a display device;
receive a selection of one or more search algorithms of a plurality of search algorithms at design time via a selection options pane, wherein the one or more search algorithms comprise at least one fuzzy selector;
and
configure an activity of an RPA workflow to for an RPA robot to execute the selected one or more search algorithms comprising the at least one fuzzy selector at design time, wherein
the one or more search algorithms comprise a selector search, a fuzzy selector search performed via the at least one fuzzy selector, a target and anchor search, a fuzzy target and anchor search, image matching, text matching, computer vision (CV) matching, or a combination thereof, and
an accuracy of the at least one fuzzy selector is configurable via the selection options pane.

6 more elements

Dependencies are inferred from the claim text.

Graphical element search technique selection, fuzzy logic selection of anchors and targets, and/or hierarchical graphical element identification for robotic process automation

Life of this patent

Graphical element search technique selection, fuzzy logic selection of anchors and targets, and/or hierarchical graphical element identification for robotic process automation

Life of this patent

Abstract

Claims29