# US 2016/0259651 A1 · Software robots for programmatically controlling computer programs to perform tasks

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2016/0259651 A1
- Document ID: US20160259651A1
- Status: Published application
- Kind: A1
- Application number: 15/059,638
- Filed: 2016-03-03
- Published: 2016-09-08
- Earliest claimed: 2015-03-03
- Assignees: Soroco Private Limited
- Applicants: Software Robotics Corporation Limited; Software Robotics Corporation
- Inventors: George Peter Nychis; Rohan Narayan Murty
- Web page: https://patentgrep.com/patent/US20160259651A1

## CPC

- G06F 3/0481: based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
- G06F 3/048: Interaction techniques based on graphical user interfaces [GUI]
- G06F 3/0482: Interaction with lists of selectable items, e.g. menus
- G06F 3/0484: for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
- G06F 3/04842: Selection of displayed objects or displayed text elements (G06F3/0482 takes precedence)
- G06F 8/38: for implementing user interfaces
- G06F 9/44: Arrangements for executing specific programs
- G06F 9/4488: {Object-oriented}
- G06F 9/451: Execution arrangements for user interfaces
- G06F 9/455: Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines
- G06F 9/45508: {Runtime interpretation or emulation, e g. emulator loops, bytecode interpretation}
- G06F 9/45512: {Command shells}
- G06F 9/45533: {Hypervisors; Virtual machine monitors}
- G06F 9/45558: {Hypervisor-specific management and integration aspects}
- G06F 11/0706: {the processing taking place on a specific hardware platform or in a specific software environment}
- G06F 11/0766: {Error or fault reporting or storing}
- G06F 11/32: with visual {or acoustical} indication of the functioning of the machine
- G06F 11/36: Prevention of errors by analysis, debugging or testing of software
- G06F 11/3612: {by runtime analysis (performance monitoring G06F11/3466)}
- G06F 11/3688: {for test execution, e.g. scheduling of test suites}
- G06F 16/9535: Search customisation based on user profiles and personalisation
- G06F 16/9538: Presentation of query results
- G06F 11/3409: {for performance assessment}
- G06F 11/3438: {monitoring of user actions (tracking the activity of the user H04L67/535)}
- G06F 11/3466: {Performance evaluation by tracing or monitoring}
- G06F 11/3476: {Data logging (G06F11/14, G06F11/2205 take precedence)}
- G06F 2009/4557: {Distribution of virtual machine instances; Migration and load balancing}
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

A system comprising at least one hardware processor configured to perform: accessing a software robot computer program for controlling at least one application program to perform a task comprising a first sub-task to be performed by a first application program; generating an object hierarchy comprising a plurality of objects corresponding to active graphical user interface (GUI) elements of the first application program; and controlling the first application program to perform the first sub-task. The controlling includes identifying, using the software robot computer program, a first action to perform in furtherance of the first sub-task; automatically accessing, in the object hierarchy, a first object corresponding to a first active GUI element of the first application program, the accessing comprising refreshing the object hierarchy; and automatically using the first object to cause the first application program to at least partially perform the first action.

## Claims

### Claim 1

A system, comprising:
at least one computer hardware processor; and
at least one non-transitory computer-readable storage medium storing:
a software robot computer program for controlling multiple application programs to perform a task comprising a first sub-task to be performed by a first application program and a second sub-task to be performed by a second application program; and
processor-executable instructions that, when executed by the at least one computer hardware processor, cause the at least one computer hardware processor to perform:
accessing the software robot computer program;
generating an object hierarchy comprising a plurality of objects corresponding to active graphical user interface (GUI) elements of at least one of the multiple application programs;
controlling the first application program, via the software robot computer program, to perform the first sub-task at least in part by:
automatically accessing, in the object hierarchy, a first object corresponding to a first active GUI element of the first application program, the accessing comprising refreshing the object hierarchy; and
automatically using the first object to cause the first application program to at least partially perform a first action in furtherance of the first sub-task; and
controlling the second application program, via the software robot computer program, to perform the second sub-task at least in part by:
automatically accessing, in the object hierarchy, a second object corresponding to a second active GUI element of the second application program, the accessing comprising refreshing the object hierarchy; and
automatically using the second object to cause the second application program to at least partially perform a second action in furtherance of the second sub-task.

### Claim 2 (depends on 1)

The system of claim 1, wherein refreshing the object hierarchy comprises:
refreshing at least one object in the object hierarchy, adding at least one object to the object hierarchy, and/or removing at least one objects from the object hierarchy.

### Claim 3 (depends on 1)

The system of claim 1, wherein refreshing the object hierarchy comprises:
refreshing only a subset of objects in the object hierarchy.

### Claim 4 (depends on 1)

The system of claim 1, wherein refreshing the object hierarchy comprises:
refreshing only one or more parent objects of the first object in the object hierarchy, one or more descendant objects of the first object in the object hierarchy, and/or the first object.

### Claim 5 (depends on 1)

The system of claim 1, wherein refreshing the object hierarchy comprises:
adding a new object to the object hierarchy, wherein the new object is associated with an active GUI element of the first application program.

### Claim 6 (depends on 1)

The system of claim 1, wherein the first active GUI element of the first application program is a visible GUI element of the first application program.

### Claim 7 (depends on 1)

The system of claim 1, wherein the first active GUI element of the first application program is a hidden GUI element of the first application program.

### Claim 8

A system, comprising:
at least one hardware processor; and
at least one non-transitory computer-readable storage medium storing:
a software robot computer program for controlling at least one application program to perform a task comprising a first sub-task to be performed by a first application program; and
processor-executable instructions that, when executed by the at least one hardware processor, cause the at least one hardware processor to perform:
accessing the software robot computer program;
generating an object hierarchy comprising a plurality of objects corresponding to active graphical user interface (GUI) elements of the first application program; and
controlling the first application program to perform the first sub-task at least in part by:
identifying, using the software robot computer program, a first action to perform in furtherance of the first sub-task;
automatically accessing, in the object hierarchy, a first object corresponding to a first active GUI element of the first application program, the accessing comprising refreshing the object hierarchy; and
automatically using the first object to cause the first application program to at least partially perform the first action.

### Claim 9 (depends on 8)

The system of claim 8, wherein the task further comprises a second sub-task to be performed by a second application program, and wherein the processor-executable instructions further cause the at least one hardware processor to control the second application program to perform the second sub-task at least in part by:
identifying, using the software robot computer program, a second action to perform in furtherance of the second sub-task;
automatically accessing, in the object hierarchy, a second object corresponding to a second active GUI element of the second application program, the accessing comprising refreshing the object hierarchy; and
automatically using the second object to cause the second application program to at least partially perform the second action.

### Claim 10 (depends on 9)

The system of claim 9, wherein the task further comprises a third sub-task to be performed by a third application program, and wherein the processor-executable instructions further cause the at least one hardware processor to control the third application program to perform the third sub-task at least in part by:
identifying, using the software robot computer program, a third action to perform in furtherance of the third sub-task;
automatically accessing, in the object hierarchy, a third object corresponding to a third active GUI element of the third application program, the accessing comprising refreshing the object hierarchy; and
automatically using the third object to cause the third application program to at least partially perform the second action.

### Claim 11 (depends on 8)

The system of claim 8, wherein refreshing the object hierarchy comprises refreshing only a subset of objects in the object hierarchy.

### Claim 12 (depends on 8)

The system of claim 8, wherein refreshing the object hierarchy comprises refreshing only the first object and/or one or more descendant objects of the first object in the object hierarchy.

### Claim 13 (depends on 8)

The system of claim 8, wherein refreshing the object hierarchy comprises adding to the object hierarchy a new object associated with an active GUI element of the first application program.

### Claim 14 (depends on 8)

The system of claim 8, wherein refreshing the object hierarchy comprises removing a particular object from the object hierarchy, wherein the particular object is associated with an inactive GUI element of the first application program.

### Claim 15 (depends on 8)

The system of claim 8, wherein refreshing the object hierarchy comprises updating at least one property of at least one object in the object hierarchy.

### Claim 16 (depends on 8)

The system of claim 8, wherein refreshing the object hierarchy comprises:
determining that a particular object in the object hierarchy is invalid; and
refreshing the particular object using a reverse tree traversal technique.

### Claim 17 (depends on 8)

The system of claim 8, wherein the accessing further comprises searching for the first object in the object hierarchy using a search string constructed in accordance with a search grammar, the search grammar comprising at least one token operator, at least one action operator, and at least one relationship operator.

### Claim 18 (depends on 8)

The system of claim 8, wherein refreshing the object hierarchy comprises:
refreshing a plurality of objects in the object hierarchy, wherein each of the plurality of objects is a descendant of the first object, and wherein refreshing the plurality of objects is performed using a plurality of threads.

### Claim 19 (depends on 18)

The system of claim 18, wherein the plurality of objects includes a first child object of the first object and a second child object of the first object, and wherein refreshing the plurality of objects comprises:
refreshing objects in the object hierarchy that are descendants of the first child object using a first thread in the plurality of threads; and
refreshing objects in the object hierarchy that are descendants of the second child object using a second thread in the plurality of threads.

### Claim 20 (depends on 19)

The system of claim 19, wherein the object hierarchy includes a first set of objects associated with the first application program and a second set of objects associated with a second application program, wherein the first set of objects includes more objects than that of the second set of objects, and wherein refreshing objects in the object hierarchy using the plurality of threads comprises using more threads in the plurality of threads to refresh objects in the first set of objects than that to refresh objects in the second set of objects.
