# US 2016/0259653 A1 · Software robots for programmatically controlling computer programs to perform tasks

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2016/0259653 A1
- Document ID: US20160259653A1
- Status: Published application
- Kind: A1
- Application number: 15/059,770
- Filed: 2016-03-03
- Published: 2016-09-08
- Earliest claimed: 2015-03-03
- Assignees: Soroco Private Limited
- Applicants: Software Robotics Corporation Limited; Software Robotics Corporation
- Inventors: George Peter Nychis; Rohan Narayan Murty
- Web page: https://patentgrep.com/patent/US20160259653A1

## CPC

- G06F 3/0481: based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
- G06F 3/048: Interaction techniques based on graphical user interfaces [GUI]
- G06F 3/0482: Interaction with lists of selectable items, e.g. menus
- G06F 3/0484: for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
- G06F 3/04842: Selection of displayed objects or displayed text elements (G06F3/0482 takes precedence)
- G06F 8/38: for implementing user interfaces
- G06F 9/44: Arrangements for executing specific programs
- G06F 9/4488: {Object-oriented}
- G06F 9/451: Execution arrangements for user interfaces
- G06F 9/455: Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines
- G06F 9/45508: {Runtime interpretation or emulation, e g. emulator loops, bytecode interpretation}
- G06F 9/45512: {Command shells}
- G06F 9/45533: {Hypervisors; Virtual machine monitors}
- G06F 9/45558: {Hypervisor-specific management and integration aspects}
- G06F 11/0706: {the processing taking place on a specific hardware platform or in a specific software environment}
- G06F 11/0766: {Error or fault reporting or storing}
- G06F 11/32: with visual {or acoustical} indication of the functioning of the machine
- G06F 11/36: Prevention of errors by analysis, debugging or testing of software
- G06F 11/3612: {by runtime analysis (performance monitoring G06F11/3466)}
- G06F 11/3688: {for test execution, e.g. scheduling of test suites}
- G06F 16/9535: Search customisation based on user profiles and personalisation
- G06F 16/9538: Presentation of query results
- G06F 11/3409: {for performance assessment}
- G06F 11/3438: {monitoring of user actions (tracking the activity of the user H04L67/535)}
- G06F 11/3466: {Performance evaluation by tracing or monitoring}
- G06F 11/3476: {Data logging (G06F11/14, G06F11/2205 take precedence)}
- G06F 2009/4557: {Distribution of virtual machine instances; Migration and load balancing}
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

A system comprising at least one computer hardware processor; and at least one computer-readable storage medium storing processor-executable instructions that, when executed by the at least one computer hardware processor, cause the at least one computer hardware processor to perform: generating an object hierarchy comprising a plurality of objects corresponding to active graphical user interface (GUI) elements of at least one of a first application program and a second application program; controlling the first application program to perform a first sub-task of a task at least in part by using the object hierarchy; and controlling the second application program to perform a second sub-task of the task at least in part by using the object hierarchy, wherein the first and second application programs are configured to execute at least in part by using different GUI application libraries.

## Claims

### Claim 1

A system, comprising:
at least one computer hardware processor; and
at least one computer-readable storage medium storing:
a software robot computer program for controlling multiple application programs to perform a task comprising a first sub-task to be performed by a first application program and a second sub-task to be performed by a second application program; and
processor-executable instructions that, when executed by the at least one computer hardware processor, cause the at least one computer hardware processor to perform:
accessing the software robot;
generating an object hierarchy comprising a plurality of objects corresponding to active graphical user interface (GUI) elements of at least one of the multiple application programs;
controlling the first application program to perform the first sub-task at least in part by using the software robot computer program and the object hierarchy; and
controlling the second application program to perform the second sub-task at least in part by using software robot computer program and the object hierarchy,
wherein the first and second application programs are configured to execute at least in part by using different GUI application libraries.

### Claim 2 (depends on 1)

The system of claim 1, wherein, when the controlling of the second application is performed, the object hierarchy comprises a first plurality of objects corresponding to a respective plurality of active GUI elements of the first application program and a second plurality of objects corresponding to a respective plurality of active GUI elements of the second application program.

### Claim 3 (depends on 1)

The system of claim 1, wherein the object hierarchy comprises a first plurality of objects corresponding to a respective plurality of active GUI elements of the first application program and a second plurality of objects corresponding to a respective plurality of active GUI elements of the second application program.

### Claim 4 (depends on 1)

The system of claim 1, wherein controlling the first application program to perform the first sub-task comprises:
automatically accessing a first object in the object hierarchy corresponding to a first active GUI element of the first application program; and
automatically using the first object to cause the first application to perform an action in furtherance of the first sub-task.

### Claim 5 (depends on 4)

The system of claim 4, wherein controlling the second application program to perform the second sub-task comprises:
automatically accessing a second object in the object hierarchy corresponding to a second active GUI element of the second application program; and
automatically using the second object to cause the second application to perform an action in furtherance of the second sub-task.

### Claim 6 (depends on 4)

The system of claim 4, wherein:
automatically accessing the first object comprises searching for the first object in the object hierarchy using a search grammar; and
automatically accessing the second object comprises searching for the second object in the object hierarchy using the search grammar.

### Claim 7 (depends on 1)

The system of claim 1, wherein the first and second application programs are configured to execute using different operating systems.

### Claim 8 (depends on 1)

The system of claim 1, wherein the first application program is a Windows application program configured to execute on a Windows operating system and the second application program is a web-based application program configured to execute at least in part by using a web-browser application program.

### Claim 9 (depends on 1)

The system of claim 1, wherein the first application program is a Windows application program and the second application program is a Java application program.

### Claim 10 (depends on 1)

The system of claim 1, wherein the first application program is an application program selected from the group consisting of an application program configured to execute on a mobile device, an application program configured to execute on a server, a Windows application program, a LINUX GTK application program, an application program having component object model (COM) access, a Java application program, a FLASH application program, a SILVERLIGHT application program, a SAP application program, an Android application program, an Oracle application program, a web-based application program, an HTML application program, a Javascript application program, an AJAX application program, a Windows MOBILE application program, an IOS application program, a SOLARIS application program, an IBM AIX application program, and a proxy application program that may be used as a proxy for controlling another application program.

### Claim 11 (depends on 1)

The system of claim 1, wherein the first and second application programs are configured to execute at least in part by using different software platforms.

### Claim 12 (depends on 11)

The system of claim 11, wherein the different software platforms include a first software platform selected from the group consisting of a Windows operating system, a LINUX operating system, a Java virtual machine, a web-browser, an operating system configured to execute on a mobile device, a SAP platform, an Oracle platform.

### Claim 13

A system, comprising:
at least one computer hardware processor; and
at least one computer-readable storage medium storing processor-executable instructions that, when executed by the at least one computer hardware processor, cause the at least one computer hardware processor to perform:
generating an object hierarchy comprising a plurality of objects corresponding to active graphical user interface (GUI) elements of at least one of a first application program and a second application program;
controlling the first application program to perform a first sub-task of a task at least in part by using the object hierarchy; and
controlling the second application program to perform a second sub-task of the task at least in part by using the object hierarchy,
wherein the first and second application programs are configured to execute at least in part by using different GUI application libraries.

### Claim 14 (depends on 13)

The system of claim 13, wherein the first and second application programs are configured to execute using different operating systems.

### Claim 15 (depends on 13)

The system of claim 13, wherein the first application program is a Windows application program configured to execute on a Windows operating system and the second application program is a web-based application program configured to execute at least in part by using a web-browser application program.

### Claim 16 (depends on 13)

The system of claim 13, wherein the first application program is a Windows application program and the second application program is a Java application program.

### Claim 17 (depends on 13)

The system of claim 13, wherein the first application program is an application program selected from the group consisting of an application program configured to execute on a mobile device, an application program configured to execute on a server, a Windows application program, a LINUX GTK application program, an application program having component object model (COM) access, a Java application program, a FLASH application program, a SILVERLIGHT application program, a SAP application program, an Android application program, an Oracle application program, a web-based application program, an HTML application program, a Javascript application program, an AJAX application program, a Windows MOBILE application program, an IOS application program, a SOLARIS application program, an IBM AIX application program, and a proxy application program that may be used as a proxy for controlling another application program.

### Claim 18 (depends on 13)

The system of claim 13, wherein the first and second application programs are configured to execute at least in part by using different software platforms.

### Claim 19 (depends on 18)

The system of claim 18, wherein the different software platforms include a first software platform selected from the group consisting of a Windows operating system, a LINUX operating system, a Java virtual machine, a web-browser, an operating system configured to execute on a mobile device, a SAP platform, an Oracle platform.
