# US 2016/0259652 A1 · Software robots for programmatically controlling computer programs to perform tasks > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2016/0259652 A1 - Document ID: US20160259652A1 - Status: Published application - Kind: A1 - Application number: 15/059,701 - Filed: 2016-03-03 - Published: 2016-09-08 - Earliest claimed: 2015-03-03 - Assignees: Soroco Private Limited - Applicants: Software Robotics Corporation Limited; Software Robotics Corporation - Inventors: George Peter Nychis; Rohan Narayan Murty - Web page: https://patentgrep.com/patent/US20160259652A1 ## CPC - G06F 3/0481: based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance - G06F 3/048: Interaction techniques based on graphical user interfaces [GUI] - G06F 3/0482: Interaction with lists of selectable items, e.g. menus - G06F 3/0484: for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range - G06F 3/04842: Selection of displayed objects or displayed text elements (G06F3/0482 takes precedence) - G06F 8/38: for implementing user interfaces - G06F 9/44: Arrangements for executing specific programs - G06F 9/4488: {Object-oriented} - G06F 9/451: Execution arrangements for user interfaces - G06F 9/455: Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines - G06F 9/45508: {Runtime interpretation or emulation, e g. emulator loops, bytecode interpretation} - G06F 9/45512: {Command shells} - G06F 9/45533: {Hypervisors; Virtual machine monitors} - G06F 9/45558: {Hypervisor-specific management and integration aspects} - G06F 11/0706: {the processing taking place on a specific hardware platform or in a specific software environment} - G06F 11/0766: {Error or fault reporting or storing} - G06F 11/32: with visual {or acoustical} indication of the functioning of the machine - G06F 11/36: Prevention of errors by analysis, debugging or testing of software - G06F 11/3612: {by runtime analysis (performance monitoring G06F11/3466)} - G06F 11/3688: {for test execution, e.g. scheduling of test suites} - G06F 16/9535: Search customisation based on user profiles and personalisation - G06F 16/9538: Presentation of query results - G06F 11/3409: {for performance assessment} - G06F 11/3438: {monitoring of user actions (tracking the activity of the user H04L67/535)} - G06F 11/3466: {Performance evaluation by tracing or monitoring} - G06F 11/3476: {Data logging (G06F11/14, G06F11/2205 take precedence)} - G06F 2009/4557: {Distribution of virtual machine instances; Migration and load balancing} - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract A system, comprising at least one computer hardware processor; and at least one non-transitory computer-readable storage medium storing processor-executable instructions that, when executed by the at least one computer hardware processor, cause the at least one computer hardware processor to perform: generating an object hierarchy comprising a plurality of objects corresponding to active graphical user interface (GUI) elements of a first application program; and controlling the first application program to perform a first sub-task. The controlling includes automatically accessing, in the object hierarchy, a first object corresponding to an active GUI element of the first application program, the accessing comprising searching for the first object in the object hierarchy, the searching comprising modifying the object hierarchy; and automatically using the first object to cause the first application program to at least partially perform a first action in furtherance of the first sub-task. ## Claims ### Claim 1 A system, comprising: at least one computer hardware processor; and at least one non-transitory computer-readable storage medium storing: a software robot computer program for controlling multiple application programs to perform a task comprising a first sub-task to be performed by a first application program and a second sub-task to be performed by a second application program; and processor-executable instructions that, when executed by the at least one computer hardware processor, cause the at least one computer hardware processor to perform: accessing the software robot computer program; generating an object hierarchy comprising a plurality of objects corresponding to active graphical user interface (GUI) elements of at least one of the multiple application programs; controlling the first application program, via the software robot computer program, to perform the first sub-task at least in part by: automatically accessing, in the object hierarchy, a first object corresponding to a first active GUI element of the first application program, the accessing comprising searching for the first object in the object hierarchy using a first search string formatted in accordance with a search grammar; and automatically using the first object to cause the first application program to at least partially perform a first action in furtherance of the first sub-task; and controlling the second application program, via the software robot computer program, to perform the second sub-task at least in part by: automatically accessing, in the object hierarchy, a second object corresponding to a second active GUI element of the second application program, the accessing comprising searching for the second object in the object hierarchy using a second search string formatted in accordance with the search grammar; and automatically using the second object to cause the second application program to at least partially perform a second action in furtherance of the second sub-task. ### Claim 2 (depends on 1) The system of claim 1, wherein the search grammar comprises a token operator, an action operator, and a relationship operator. ### Claim 3 (depends on 2) The system of claim 2, wherein the first search string includes the action operator. ### Claim 4 (depends on 3) The system of claim 3, wherein the action operator indicates how the object hierarchy is to be modified during performance of the search. ### Claim 5 (depends on 1) The system of claim 1, wherein searching for the first object in the object hierarchy comprises modifying the object hierarchy. ### Claim 6 (depends on 1) The system of claim 1, wherein searching for the first object further comprises: searching for a third object in the object hierarchy; modifying the object hierarchy by adding one or more new objects to the object hierarchy as a descendants of the third object to obtain a modified object hierarchy; and searching for the first object in the modified object hierarchy. ### Claim 7 (depends on 6) The system of claim 6, wherein modifying the object hierarchy comprises: using the third object to cause the first application program to perform an action causing one or more new GUI elements of the first application program to become active; and modifying the object hierarchy by adding one or more new objects corresponding to the one or more new GUI elements as descendants of the third object in the object hierarchy. ### Claim 8 (depends on 6) The system of claim 6, wherein: searching for the third object is performed using a search string formatted in accordance with the search grammar, the search string including an action operator. ### Claim 9 (depends on 1) The system of claim 1, wherein the first active GUI element of the first application program is a visible GUI element of the first application program. ### Claim 10 (depends on 1) The system of claim 1, wherein the first active GUI element of the first application program is a hidden GUI element of the first application program. ### Claim 11 A system, comprising: at least one computer hardware processor; and at least one non-transitory computer-readable storage medium storing: a software robot computer program for controlling at least one application program to perform a task comprising a first sub-task to be performed by a first application program; and processor-executable instructions that, when executed by the at least one computer hardware processor, cause the at least one computer hardware processor to perform: accessing the software robot computer program; generating an object hierarchy comprising a plurality of objects corresponding to active graphical user interface (GUI) elements of the at least one application program to be controlled by the software robot computer program; controlling the first application program to perform the first sub-task at least in part by: identifying, using the software robot computer program, a first action to perform in furtherance of the first sub-task; automatically accessing, in the object hierarchy, a first object corresponding to an active GUI element of the first application program, the accessing comprising searching for the first object in the object hierarchy, the searching comprising modifying the object hierarchy; and automatically using the first object to cause the first application program to at least partially perform the first action. ### Claim 12 (depends on 11) The system of claim 11, wherein searching for the first object is performed using a first search string formatted in accordance with a search grammar. ### Claim 13 (depends on 12) The system of claim 12, wherein the search grammar comprises a token operator, an action operator, and a relationship operator. ### Claim 14 (depends on 13) The system of claim 13, wherein the search string includes the action operator. ### Claim 15 (depends on 14) The system of claim 14, wherein the action operator indicates how the object hierarchy is to be modified during performance of the search. ### Claim 16 (depends on 11) The system of claim 11, wherein searching for the first object further comprises: searching for a second object in the object hierarchy; modifying the object hierarchy by adding one or more new objects to the object hierarchy as descendants of the second object to obtain a modified object hierarchy; and searching for the first object in the modified object hierarchy. ### Claim 17 (depends on 16) The system of claim 16, wherein modifying the object hierarchy comprises: using the second object to cause the first application program to perform an action causing one or more new GUI elements of the first application program to become active; and modifying the object hierarchy by adding one or more new objects corresponding to the one or more new GUI elements as descendants of the second object in the object hierarchy. ### Claim 18 (depends on 16) The system of claim 16, wherein searching for the second object is performed using a search string formatted in accordance with a search grammar, the search string including an action operator. ### Claim 19 (depends on 11) The system of claim 11, wherein the first active GUI element of the first application program is a visible GUI element of the first application program. ### Claim 20 A system, comprising: at least one computer hardware processor; and at least one non-transitory computer-readable storage medium storing processor-executable instructions that, when executed by the at least one computer hardware processor, cause the at least one computer hardware processor to perform: generating an object hierarchy comprising a plurality of objects corresponding to active graphical user interface (GUI) elements of a first application program; controlling the first application program to perform a first sub-task of a task at least in part by: automatically accessing, in the object hierarchy, a first object corresponding to an active GUI element of the first application program, the accessing comprising searching for the first object in the object hierarchy, the searching comprising modifying the object hierarchy; and automatically using the first object to cause the first application program to at least partially perform a first action in furtherance of the first sub-task.