# US 2022/0032471 A1 · Robotic process automation using enhanced object detection to provide resilient playback capabilities

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2022/0032471 A1
- Document ID: US20220032471A1
- Status: Published application
- Kind: A1
- Application number: 17/139,842
- Filed: 2020-12-31
- Published: 2022-02-03
- Earliest claimed: 2020-08-03
- Assignees: Automation Anywhere, Inc.
- Applicants: Automation Anywhere, Inc.
- Inventors: Sudhir Kumar SINGH; JESSE TRUSCOTT; VIRINCHIPURAM ANAND; HARSHIL LODHIYA
- Web page: https://patentgrep.com/patent/US20220032471A1

## CPC

- G06F 9/451: Execution arrangements for user interfaces
- G06V 30/10: Character recognition
- B25J 13/06: Control stands, e.g. consoles, switchboards
- G06F 3/0484: for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
- G06F 3/0485: Scrolling or panning
- G05B 2219/32128: Gui graphical user interface
- G05B 2219/50391: Robot
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- B25J: MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES ({manipulators specially adapted for use in surgery A61B34/70; manipulators used in cleaning hollow articles B08B9/04}; manipulators associated with rolling mills B21B39/20; manipulators associated with forging machines B21J13/10; {manipulators associated with picking-up and placing mechanisms B23P19/007}; means for holding wheels or parts thereof B60B30/00; {vehicles with ground-engaging propulsion means, e.g. walking members B62D57/02, B62D57/032; devices for picking-up and depositing articles or materials between conveyors B65G47/90, B65G47/91; manipulators with gripping or holding means for transferring packages B65H67/065}; cranes B66C; {manipulators used in the protection or supervision of pipe-line installations F17D5/00; walking equipment adapted for nuclear steam-generators F22B37/006}; manipulators specially adapted for, or associated with, nuclear reactors G21C; {apparatus used for handling wafers during manufacture or treatment of semiconductor H10P72/50})
- G05B: CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS (systems for controlling or regulating non-electric variables G05D; systems for regulating electric or magnetic variables G05F; control devices or systems insofar as characterised by mechanical features only G05G)

## Abstract

Robotic process automation (RPA) systems with improved playback capabilities. Certain embodiments can provide resilient playback of software automation processes by providing the capability to record, compute and store parameters for user interface controls detected from a screen image of a user interface. These parameters can be used to assist in locating correct corresponding user interface controls within a screen image presented at playback of a software automation process. Other embodiments can, additionally or alternatively, provide resilient playback of software automation processes by providing enhanced capability to locate user interface controls within a screen image of a user interface. In some embodiments, one or more of the user interface controls located within the screen image of the user interface can be used to manipulate the user interface so that other user interface controls become visible within the screen image. Advantageously, embodiments disclosed herein allow software automation processes to operate with greater reliability and flexibility.

## Claims

### Claim 1

A computer-implemented method for facilitating execution of a robotic process automation, the method comprising:
capturing an image of a user interface (UI) presented on a display device by an application program operating on a computing device;
detecting a plurality of UI controls within the captured image of the UI by programmatic examination of the captured image;
detecting positioning data and sizing data for each of the plurality of UI controls;
detecting a plurality of UI text labels within the captured image of the UI by programmatic examination of the captured image;
detecting positioning data and sizing data for each of the UI text labels; and
associating the UI text labels to the UI controls based on the positioning data and the sizing data for the UI text labels and for the UI controls.

### Claim 2 (depends on 1)

A computer-implemented method as recited in claim 1, wherein the associating is based on a separation distance between the UI text labels and the UI controls.

### Claim 3 (depends on 1)

A computer-implemented method as recited in claim 1, wherein the method comprises:
recording a series of user interactions with a plurality of different ones of the UI controls of the UI.

### Claim 4 (depends on 1)

A computer-implemented method as recited in claim 1, wherein a plurality of the UI controls are chosen from the group consisting of: TEXTBOX, COMBOBOX, TEXT BUTTON, CHECKBOX, and RADIO BUTTON.

### Claim 5 (depends on 1)

A computer-implemented method as recited in claim 1, wherein the detecting positioning data and sizing data for each of the UI controls is by programmatic examination of the captured image.

### Claim 6 (depends on 1)

A computer-implemented method as recited in claim 1, wherein the detecting positioning data and sizing data for each of the UI text labels is by programmatic examination of the captured image.

### Claim 7 (depends on 1)

A computer-implemented method as recited in claim 1, wherein the detecting positioning data and sizing data for each of the UI controls comprises geometry data.

### Claim 8 (depends on 7)

A computer-implemented method as recited in claim 7, wherein the detecting positioning data and sizing data for each of the UI text labels comprises geometry data.

### Claim 9 (depends on 7)

A computer-implemented method as recited in claim 7, wherein the geometry data for at least a plurality of the UI controls includes at least coordinates defining a boundary box.

### Claim 10 (depends on 1)

A computer-implemented method as recited in claim 1, wherein the captured image is a screen image.

### Claim 11

A computer-implemented method for facilitating execution of a robotic process automation, the method comprising:
capturing an initial image of an initial user interface (UI) presented on a display device by an application program operating on a computing device;
detecting a plurality of UI controls of the initial UI within the captured initial image by programmatic examination of the captured initial image;
detecting positioning data and sizing data for each of the UI controls of the initial UI;
detecting a plurality of UI text labels of the initial UI within the captured initial image by programmatic examination of the captured initial image;
detecting positioning data and sizing data for each of the UI text labels;
associating the UI text labels to the UI controls based on the positioning data and the sizing data for the UI text labels and for UI controls of the initial UI;
recording a series of user interactions with a plurality of different ones of the UI controls of the initial UI;
subsequently capturing a subsequent image of a subsequent UI presented on a display device, the subsequent UI being generated by the same application program as the application program generating the initial UI, with the same application program operating on a computing device and presenting the subsequent UI on a display device;
detecting a plurality of UI controls of the subsequent UI within the captured subsequent image by programmatic examination of the captured subsequent image;
detecting positioning data and sizing data for each of the UI controls of the subsequent UI;
detecting a plurality of UI text labels of the subsequent UI within the captured subsequent image by programmatic examination of the captured subsequent image;
detecting positioning data and sizing data for each of the UI text labels of the subsequent UI;
associating the UI text labels to the UI controls based on the positioning data and the sizing data for the UI text labels and for UI controls of the subsequent UI;
matching the UI controls of the subsequent UI to the UI controls of the initial UI, the matching using the UI text labels that are associated with the respective UI controls of the initial UI and the subsequent UI; and
programmatically causing one or more interactions of the series of user interactions previously recorded with respect to at least one of the UI controls of the initial UI to be provided to and induced on at least one of the UI controls of the subsequent UI that has been matched to the at least one of the UI controls of the initial UI.

### Claim 12 (depends on 11)

A computer-implemented method as recited in claim 11, wherein the method comprises:
manipulating the subsequent UI if none of the UI controls of the subsequent UI match the at least one of the UI controls of the initial UI.

### Claim 13 (depends on 12)

A computer-implemented method as recited in claim 12, wherein the manipulating induces a scrolling action to the subsequent UI.

### Claim 14 (depends on 11)

A computer-implemented method as recited in claim 11, wherein the matching of the UI controls of the subsequent UI to the UI controls of the initial UI is facilitated by the associated UI text labels respectively associated to the UI controls of the subsequent UI to the UI controls of the initial UI.

### Claim 15 (depends on 11)

A computer-implemented method as recited in claim 11, wherein the associating of the UI text labels to the UI controls based on separation distance and direction therebetween.

### Claim 16 (depends on 11)

A computer-implemented method as recited in claim 11, wherein the matching comprises
determining a plurality of candidate matching UI controls in the subsequent UI that potentially match a particular UI control of the UI controls of the initial UI used by a particular interaction of the series of user interactions previously recorded; and
selecting one of the candidate matching UI controls as a matching UI control.

### Claim 17 (depends on 11)

A computer-implemented method as recited in claim 11, wherein the matching comprises
determining a plurality of candidate matching UI controls in the subsequent UI that potentially match a particular UI control of the UI controls of the initial UI used by a particular interaction of the series of user interactions previously recorded;
determining a confidence value for each of the candidate matching UI controls; and
selecting one of the candidate matching UI controls as a matching UI control based on the confidence value.

### Claim 18 (depends on 11)

A computer-implemented method as recited in claim 11, wherein the matching comprises
determining a plurality of candidate matching UI controls in the subsequent UI that potentially match a particular UI control of the UI controls of the initial UI used by a particular interaction of the series of user interactions previously recorded;
determining a confidence value for each of the candidate matching UI controls;
eliminating those of the candidate matching UI controls that have a confidence value less than a predetermined threshold; and
selecting one of the candidate matching UI controls that remain after the eliminating as a matching UI control.

### Claim 19

A non-transitory computer readable medium including at least computer program code tangibly stored thereon for facilitating execution of a robotic process automation, the computer readable medium comprising:
computer program code for capturing an image of a user interface (UI) presented on a display device by an application program operating on a computing device;
computer program code for detecting a plurality of UI controls within the captured image of the UI by programmatic examination of the captured image, the detecting of the UI controls including detecting parameters of the UI controls;
computer program code for detecting a plurality of UI text labels within the captured image of the UI by programmatic examination of the captured image, the detecting of the UI text labels including detecting parameters of the UI text labels; and
computer program code for associating the UI text labels to the UI controls based on the parameters of the UI text labels and the parameters of the UI controls.

### Claim 20 (depends on 19)

A non-transitory computer readable medium as recited in claim 19, wherein the parameters include geographic position data of the UI controls and the UI text labels relative to the captured image.

### Claim 21 (depends on 19)

A non-transitory computer readable medium as recited in claim 19, wherein the computer program code for associating of the UI text labels to the UI controls is based on a separation distance and a direction from each of one or more of the UI controls to each of one or more of the UI text labels.

### Claim 22

A non-transitory computer readable medium including at least computer program code tangibly stored thereon for facilitating execution of a robotic process automation, the computer readable medium comprising:
computer program code for capturing an initial screen image of an initial user interface (UI) presented on a display device by an application program operating on a computing device;
computer program code for detecting a plurality of UI controls of the initial UI within the captured initial screen image by programmatic examination of the captured initial screen image;
computer program code for detecting positioning data and sizing data for each of the UI controls of the initial UI;
computer program code for detecting a plurality of UI text labels of the initial UI within the captured initial screen image by programmatic examination of the captured initial screen image;
computer program code for detecting positioning data and sizing data for each of the UI text labels;
computer program code for associating the UI text labels to the UI controls based on the positioning data and the sizing data for the UI text labels and for UI controls of the initial UI;
computer program code for recording a series of user interactions with a plurality of different ones of the UI controls of the initial UI;
computer program code for subsequently capturing a subsequent screen image of a subsequent UI presented on a display device, the subsequent UI being generated by the same application program as the application program generating the initial UI, with the same application program operating on a computing device and presenting the subsequent UI on a display device;
computer program code for detecting a plurality of UI controls of the subsequent UI within the captured subsequent screen image by programmatic examination of the captured subsequent screen image;
computer program code for detecting positioning data and sizing data for the UI controls of the subsequent UI;
computer program code for detecting a plurality of UI text labels of the subsequent UI within the captured subsequent screen image by programmatic examination of the captured subsequent screen image texts;
computer program code for detecting positioning data and sizing data for each of the UI text labels of the subsequent UI;
computer program code for associating the UI text labels to the UI controls based on the positioning data and the sizing data for the UI text labels and for UI controls of the subsequent UI;
computer program code for matching the UI controls of the subsequent UI to the UI controls of the initial UI, the matching using the UI text labels that are associated with the respective UI controls of the initial UI and the subsequent UI; and
computer program code for programmatically causing one or more interactions of the series of user interactions previously recorded with respect to at least one of the UI controls of the initial UI to be provided to and induced on at least one of the UI controls of the subsequent UI that has been matched to the at least one of the UI controls of the initial UI.

### Claim 23 (depends on 22)

A non-transitory computer readable medium as recited in claim 22, wherein the computer readable medium comprises:
computer program code for manipulating the subsequent UI if none of the UI controls of the subsequent UI match the at least one of the UI controls of the initial UI.
