# US 2016/0259655 A1 · Software robots for programmatically controlling computer programs to perform tasks

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2016/0259655 A1
- Document ID: US20160259655A1
- Status: Published application
- Kind: A1
- Application number: 15/059,907
- Filed: 2016-03-03
- Published: 2016-09-08
- Earliest claimed: 2015-03-03
- Assignees: Soroco Private Limited
- Applicants: Software Robotics Corporation Limited; Software Robotics Corporation
- Inventors: George Peter Nychis; Rohan Narayan Murty
- Web page: https://patentgrep.com/patent/US20160259655A1

## CPC

- G06F 3/0481: based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
- G06F 3/048: Interaction techniques based on graphical user interfaces [GUI]
- G06F 3/0482: Interaction with lists of selectable items, e.g. menus
- G06F 3/0484: for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
- G06F 3/04842: Selection of displayed objects or displayed text elements (G06F3/0482 takes precedence)
- G06F 8/38: for implementing user interfaces
- G06F 9/44: Arrangements for executing specific programs
- G06F 9/4488: {Object-oriented}
- G06F 9/451: Execution arrangements for user interfaces
- G06F 9/455: Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines
- G06F 9/45508: {Runtime interpretation or emulation, e g. emulator loops, bytecode interpretation}
- G06F 9/45512: {Command shells}
- G06F 9/45533: {Hypervisors; Virtual machine monitors}
- G06F 9/45558: {Hypervisor-specific management and integration aspects}
- G06F 11/0706: {the processing taking place on a specific hardware platform or in a specific software environment}
- G06F 11/0766: {Error or fault reporting or storing}
- G06F 11/32: with visual {or acoustical} indication of the functioning of the machine
- G06F 11/36: Prevention of errors by analysis, debugging or testing of software
- G06F 11/3612: {by runtime analysis (performance monitoring G06F11/3466)}
- G06F 11/3688: {for test execution, e.g. scheduling of test suites}
- G06F 16/9535: Search customisation based on user profiles and personalisation
- G06F 16/9538: Presentation of query results
- G06F 11/3409: {for performance assessment}
- G06F 11/3438: {monitoring of user actions (tracking the activity of the user H04L67/535)}
- G06F 11/3466: {Performance evaluation by tracing or monitoring}
- G06F 11/3476: {Data logging (G06F11/14, G06F11/2205 take precedence)}
- G06F 2009/4557: {Distribution of virtual machine instances; Migration and load balancing}
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

A system comprising at least one hardware processor; at least one non-transitory computer-readable storage medium storing processor-executable instructions that, when executed by the at least one hardware processor, cause the at least one hardware processor to perform: generating an object hierarchy comprising a plurality of objects corresponding to active graphical user interface (GUI) elements of a first application program, the generating comprising: obtaining a hierarchical representation of the active GUI elements of the application program; generating an object for each of at least some of the active GUI elements represented in the hierarchical representation; and organizing the generated objects into the object hierarchy based on relationships among active GUI elements represented in the hierarchical representation; and controlling, using software robot computer program and the generated object hierarchy, the first application program to perform the first sub-task.

## Claims

### Claim 1

A system, comprising:
at least one hardware processor;
at least one non-transitory computer-readable storage medium storing:
a software robot computer program for controlling multiple application programs to perform a task comprising a first sub-task to be performed by a first application program and a second sub-task to be performed by a second application program; and
processor-executable instructions that, when executed by the at least one hardware processor, cause the at least one hardware processor to perform:
accessing the software robot computer program;
generating an object hierarchy comprising a plurality of objects corresponding to active graphical user interface (GUI) elements of the first and second application programs, the generating comprising:
generating, based on a first hierarchical representation of active GUI elements of the first application, a first portion of the object hierarchy corresponding to active GUI elements of the first application program;
generating, based on a second hierarchical representation of active GUI elements of the second application, a second portion of the object hierarchy corresponding to the active GUI elements of the second application program, and
combining the first portion and the second portion; and
controlling, using software robot computer program and the generated object hierarchy, the first application program to perform the first sub-task and the second application program to perform the second sub-task.

### Claim 2 (depends on 1)

The system of claim 1, wherein the processor-executable instructions further cause the at least one hardware processor to obtain the first hierarchical representation of the active GUI elements of the first application program from an operating system.

### Claim 3 (depends on 2)

The system of claim 2, wherein the processor-executable instructions further cause the at least one hardware processor to obtain the second hierarchical representation from the second application program.

### Claim 4 (depends on 1)

The system of claim 1, wherein the processor-executable instructions further cause the at least one hardware processor to obtain the first hierarchical representation by using the Windows Automation API.

### Claim 5 (depends on 1)

The system of claim 1, wherein the first application program is an application program implemented using a Windows GUI application library and the second application program is an Internet browser program, and wherein the processor-executable instructions further cause the at least one computer hardware processor to perform:
obtaining the first hierarchical representation via the Windows Automation API; and obtaining the second hierarchical representation via a Document Object Model (DOM) representation of a webpage displayed by the Internet browser program.

### Claim 6 (depends on 1)

The system of claim 1, wherein the first and second application programs are configured to execute at least in part by using different GUI application libraries.

### Claim 7 (depends on 1)

The system of claim 1, wherein controlling the first application program to perform the first sub-task comprises:
accessing, in the object hierarchy, a first object corresponding to a first active GUI element of the first application program; and
using the first object to cause the first application program to at least partially perform a first action in furtherance of the first sub-task.

### Claim 8 (depends on 7)

The system of claim 7, wherein accessing the first object comprises refreshing the object hierarchy.

### Claim 9

A system, comprising:
at least one hardware processor;
at least one non-transitory computer-readable storage medium storing processor-executable instructions that, when executed by the at least one hardware processor, cause the at least one hardware processor to perform:
generating an object hierarchy comprising a plurality of objects corresponding to active graphical user interface (GUI) elements of a first application program, the generating comprising:
obtaining a hierarchical representation of the active GUI elements of the application program;
generating an object for each of at least some of the active GUI elements represented in the hierarchical representation; and
organizing the generated objects into the object hierarchy based on relationships among active GUI elements represented in the hierarchical representation; and
controlling, using software robot computer program and the generated object hierarchy, the first application program to perform the first sub-task.

### Claim 10 (depends on 9)

The system of claim 9, wherein the processor-executable instructions further cause the at least one hardware processor to obtain the hierarchical representation of the active GUI elements of the first application program from an operating system.

### Claim 11 (depends on 9)

The system of claim 9, wherein the processor-executable instructions further cause the at least one hardware processor to obtain the hierarchical representation of the active GUI elements of the first application program via the Windows Automation API.

### Claim 12 (depends on 9)

The system of claim 9, wherein controlling the first application program to perform the first sub-task comprises:
accessing, in the object hierarchy, a first object corresponding to a first active GUI element of the first application program; and
using the first object to cause the first application program to at least partially perform a first action in furtherance of the first sub-task.

### Claim 13 (depends on 9)

The system of claim 9, wherein accessing the first object comprises refreshing the object hierarchy.

### Claim 14

A system for controlling multiple application programs to perform a task comprising a first sub-task and a second sub-task, the system comprising:
at least one hardware processor;
at least one non-transitory computer-readable storage medium storing processor-executable instructions that, when executed by the at least one hardware processor, cause the at least one hardware processor to perform:
generating an object hierarchy comprising a plurality of objects corresponding to active graphical user interface (GUI) elements of a first application program and a second application program, the generating comprising:
generating, based on a first hierarchical representation of active GUI elements of the first application program, a first portion of the object hierarchy corresponding to active GUI elements of the first application program;
generating, based on a second hierarchical representation of active GUI elements of the second application program, a second portion of the object hierarchy corresponding to the active GUI elements of the second application program, and
combining the first portion and the second portion; and
controlling, using the generated object hierarchy, the first and second application program, respectively, to perform the first sub-task and the second sub-task.

### Claim 15 (depends on 14)

The system of claim 14, wherein the processor-executable instructions further cause the at least one hardware processor to obtain the first hierarchical representation of the active GUI elements of the first application program from an operating system.

### Claim 16 (depends on 15)

The system of claim 15, wherein the processor-executable instructions further cause the at least one hardware processor to obtain the second hierarchical representation from the second application program.

### Claim 17 (depends on 14)

The system of claim 14, wherein the processor-executable instructions further cause the at least one hardware processor to obtain the first hierarchical representation via the Windows Automation API.

### Claim 18 (depends on 14)

The system of claim 14, wherein the first application program is an application program implemented using a Windows GUI application library and the second application program is an Internet browser program, and wherein the processor-executable instructions further cause the at least one computer hardware processor to perform:
obtaining the first hierarchical representation via the Windows Automation API; and
obtaining the second hierarchical representation via a Document Object Model (DOM) representation of a webpage displayed by the Internet browser program.
