# US 10,310,701 B2 · Software robots for programmatically controlling computer programs to perform tasks > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,310,701 B2 - Document ID: US10310701B2 - Status: Granted - Kind: B2 - Application number: 15/942,930 - Filed: 2018-04-02 - Published: 2018-08-09 - Granted: 2019-06-04 - Earliest claimed: 2015-03-03 - Assignees: Soroco Private Limited - Applicants: Soroco Private Limited - Inventors: George Peter Nychis; Rohan Narayan Murty - Web page: https://patentgrep.com/patent/US10310701B2 ## CPC - G06F 3/0481: based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance - G06F 3/048: Interaction techniques based on graphical user interfaces [GUI] - G06F 3/0482: Interaction with lists of selectable items, e.g. menus - G06F 3/0484: for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range - G06F 3/04842: Selection of displayed objects or displayed text elements (G06F3/0482 takes precedence) - G06F 8/38: for implementing user interfaces - G06F 9/44: Arrangements for executing specific programs - G06F 9/4488: {Object-oriented} - G06F 9/451: Execution arrangements for user interfaces - G06F 9/455: Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines - G06F 9/45508: {Runtime interpretation or emulation, e g. emulator loops, bytecode interpretation} - G06F 9/45512: {Command shells} - G06F 9/45533: {Hypervisors; Virtual machine monitors} - G06F 9/45558: {Hypervisor-specific management and integration aspects} - G06F 11/0706: {the processing taking place on a specific hardware platform or in a specific software environment} - G06F 11/0766: {Error or fault reporting or storing} - G06F 11/32: with visual {or acoustical} indication of the functioning of the machine - G06F 11/36: Prevention of errors by analysis, debugging or testing of software - G06F 11/3612: {by runtime analysis (performance monitoring G06F11/3466)} - G06F 11/3688: {for test execution, e.g. scheduling of test suites} - G06F 16/9535: Search customisation based on user profiles and personalisation - G06F 16/9538: Presentation of query results - G06F 11/3409: {for performance assessment} - G06F 11/3438: {monitoring of user actions (tracking the activity of the user H04L67/535)} - G06F 11/3466: {Performance evaluation by tracing or monitoring} - G06F 11/3476: {Data logging (G06F11/14, G06F11/2205 take precedence)} - G06F 2009/4557: {Distribution of virtual machine instances; Migration and load balancing} - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract A system comprising at least one computer hardware processor configured to perform: generating an object hierarchy comprising a plurality of objects corresponding to active graphical user interface (GUI) elements of at least one application program; controlling the at least one application program to perform a task comprising a sequence of actions at least in part by using the object hierarchy to invoke actions in the sequence of actions. ## Claims ### Claim 1 A system for controlling at least one application program for performing a task comprising a sequence of actions, the at least one application program including a first application program and a second application program, the first and second application programs configured to execute at least in part by using different graphical user interface (GUI) application libraries, the system comprising: at least one computer hardware processor; and at least one non-transitory computer-readable storage medium storing processor-executable instructions that, when executed by the at least one computer hardware processor, cause the at least one computer hardware processor to perform: generating an object hierarchy comprising a plurality of objects corresponding to active GUI elements of the at least one application program, the generating comprising: identifying a first plurality of active GUI elements of the first application program and a second plurality of active GUI elements of the second application program, wherein the first application program is configured to execute by using a first GUI application library and the second application program is configured to execute by using a second GUI application library that is different than the first GUI application library; generating a first portion of the object hierarchy comprising a first plurality of objects corresponding to the first plurality of active GUI elements of the first application program and a second portion of the object hierarchy comprising a second plurality of objects corresponding to the second plurality of active GUI elements of the second application program, controlling the first application program to perform a first action in the sequence of actions using the first portion of the object hierarchy at least in part by: using a first object of the first plurality of objects to cause, via a first active GUI element of the first plurality of active GUI elements, the first application program to at least partially perform the first action, the first action programmatically emulating a user's interaction with the first active GUI element; and after the first application program performs the first action, refreshing the object hierarchy to obtain a first refreshed object hierarchy comprising a third plurality of objects corresponding to a third plurality of active GUI elements reflecting changes in a user interface of the first application program; and controlling the second application program to perform a second action in the sequence of actions using the second portion of the object hierarchy at least in part by: using a second object of the second plurality of objects to cause, via a second active GUI element of the second plurality of active GUI elements, the second application program to at least partially perform the second action, the second action programmatically emulating a user's interaction with the second active GUI element; and after the second application program performs the second action, refreshing the object hierarchy to obtain a second refreshed object hierarchy comprising a fourth plurality of objects corresponding to a fourth plurality of active GUI elements reflecting changes in a user interface of the second application program. ### Claim 2 (depends on 1) The system of claim 1, wherein refreshing the object hierarchy comprises: refreshing at least one object in the object hierarchy, adding at least one object to the object hierarchy, and/or removing at least one object from the object hierarchy. ### Claim 3 (depends on 1) The system of claim 1, wherein refreshing the object hierarchy comprises: refreshing only a subset of objects in the object hierarchy. ### Claim 4 (depends on 1) The system of claim 1, wherein controlling the first application program to perform the first action in the sequence of actions comprises: accessing the first object in the object hierarchy corresponding to the first active GUI element of the first application program. ### Claim 5 (depends on 1) The system of claim 1, wherein refreshing the object hierarchy comprises: refreshing only one or more parent objects of the first object in the object hierarchy, one or more descendant objects of the first object in the object hierarchy, and/or the first object. ### Claim 6 (depends on 1) The system of claim 1, wherein refreshing the object hierarchy comprises: adding a new object to the object hierarchy, wherein the new object is associated with an active GUI element of the first application program. ### Claim 7 (depends on 1) The system of claim 1, wherein the first active GUI element of the first application program is a visible GUI element of the first application program. ### Claim 8 (depends on 1) The system of claim 1, further comprising a display, wherein the processor-executable instructions further cause the at least one computer hardware processor to perform: displaying a GUI of the first application program on the display during the performance of at least some of the sequence of actions. ### Claim 9 (depends on 1) The system of claim 1, wherein generating the object hierarchy comprises: obtaining a first hierarchical representation of the first plurality of active GUI elements of the first application program from an operating system; and identifying the first plurality of active GUI elements of the first application program using the first hierarchical representation. ### Claim 10 (depends on 9) The system of claim 9, wherein obtaining the first hierarchical representation is performed by using the WINDOWS Automation API. ### Claim 11 (depends on 1) The system of claim 1, wherein the processor-executable instructions, when executed by the at least one computer hardware processor, further cause the at least one computer hardware processor to perform: controlling the first application program to perform a third action in the sequence of actions using the first refreshed object hierarchy at least in part by: using a third object of the third plurality of objects to cause, via a third active GUI element of the third plurality of active GUI elements, the first application program to at least partially perform the third action, the third action programmatically emulating a user's interaction with the third active GUI element. ### Claim 12 (depends on 1) The system of claim 1, wherein the processor-executable instructions, when executed by the at least one computer hardware processor, further cause the at least one computer hardware processor to perform: controlling the second application program to perform a fourth action in the sequence of actions using the second refreshed object hierarchy at least in part by: using a fourth object of the fourth plurality of objects to cause, via a fourth active GUI element of the fourth plurality of active GUI elements, the second application program to at least partially perform the fourth action, the fourth action programmatically emulating a user's interaction with the fourth active GUI element. ### Claim 13 (depends on 1) The system of claim 1, wherein the first and second application programs are configured to execute using different operating systems. ### Claim 14 (depends on 1) The system of claim 1, wherein the first application program is a WINDOWS application program configured to execute on a WINDOWS operating system and the second application program is a web-based application program configured to execute at least in part by using a web-browser application program. ### Claim 15 (depends on 1) The system of claim 1, wherein the first application program is a WINDOWS application program and the second application program is a JAVA application program. ### Claim 16 (depends on 1) The system of claim 1, wherein the first and second application programs are configured to execute at least in part by using different software platforms. ### Claim 17 (depends on 16) The system of claim 16, wherein the different software platforms include a first software platform selected from the group consisting of a WINDOWS operating system, a LINUX operating system, a JAVA virtual machine, a web-browser, an operating system configured to execute on a mobile device, a SAP platform, and an ORACLE platform. ### Claim 18 A method for controlling at least one application program for performing a task comprising a sequence of actions, the at least one application program including a first application program and a second application program, the first and second application programs configured to execute at least in part by using different graphical user interface (GUI) application libraries, the method comprising: using at least one computer hardware processor to perform: generating an object hierarchy comprising a plurality of objects corresponding to active GUI elements of the at least one application program, the generating comprising: identifying a first plurality of active GUI elements of the first application program and a second plurality of active GUI elements of the second application program, wherein the first application program is configured to execute by using a first GUI application library and the second application program is configured to execute by using a second GUI application library that is different than the first GUI application library; generating a first portion of the object hierarchy comprising a first plurality of objects corresponding to the first plurality of active GUI elements of the first application program and a second portion of the object hierarchy comprising a second plurality of objects corresponding to the second plurality of active GUI elements of the second application program; controlling the first application program to perform a first action in the sequence of actions using the first portion of the object hierarchy at least in part by: using a first object of the first plurality of objects to cause, via a first active GUI element of the first plurality of active GUI elements, the first application program to at least partially perform the first action, the first action programmatically emulating a user's interaction with the first active GUI element; and after the first application program performs the first action, refreshing the object hierarchy to obtain a first refreshed object hierarchy comprising a third plurality of objects corresponding to a third plurality of active GUI elements reflecting changes in a user interface of the first application program; and controlling the second application program to perform a second action in the sequence of actions using the second portion of the object hierarchy at least in part by: using a second object of the second plurality of objects to cause, via a second active GUI element of the second plurality of active GUI elements, the second application program to at least partially perform the second action, the second action programmatically emulating a user's interaction with the second active GUI element; and after the second application program performs the second action, refreshing the object hierarchy to obtain a second refreshed object hierarchy comprising a fourth plurality of objects corresponding to a fourth plurality of active GUI elements reflecting changes in a user interface of the second application program. ### Claim 19 (depends on 18) The method of claim 18, wherein refreshing the object hierarchy comprises: refreshing at least one object in the object hierarchy, adding at least one object to the object hierarchy, and/or removing at least one object from the object hierarchy. ### Claim 20 (depends on 18) The method of claim 18, wherein refreshing the object hierarchy comprises: refreshing only a subset of objects in the object hierarchy. ### Claim 21 (depends on 18) The method of claim 18, wherein controlling the first application program to perform the first action in the sequence of actions comprises: accessing the first object in the object hierarchy corresponding to the first active GUI element of the first application program. ### Claim 22 (depends on 21) The method of claim 21, wherein refreshing the object hierarchy comprises: refreshing only one or more parent objects of the first object in the object hierarchy, one or more descendant objects of the first object in the object hierarchy, and/or the first object. ### Claim 23 (depends on 18) The method of claim 18, wherein generating the object hierarchy comprises: obtaining a first hierarchical representation of the first plurality of active GUI elements of the first application program from an operating system; and identifying the first plurality of active GUI elements of the first application program using the first hierarchical representation. ### Claim 24 At least one non-transitory computer-readable storage medium storing processor-executable instructions that, when executed by at least one computer hardware processor, causes the at least one computer hardware processor to perform a method for controlling at least one application program for performing a task comprising a sequence of actions, the at least one application program including a first application program and a second application program, the first and second application programs configured to execute at least in part by using different graphical user interface (GUI) application libraries, the method comprising: generating an object hierarchy comprising a plurality of objects corresponding to active GUI elements of the at least one application program, the generating comprising: identifying a first plurality of active GUI elements of the first application program and a second plurality of active GUI elements of the second application program, wherein the first application program is configured to execute by using a first GUI application library and the second application program is configured to execute by using a second GUI application library that is different than the first GUI application library; generating a first portion of the object hierarchy comprising a first plurality of objects corresponding to the first plurality of active GUI elements of the first application program and a second portion of the object hierarchy comprising a second plurality of objects corresponding to the second plurality of active GUI elements of the second application program; controlling the first application program to perform a first action in the sequence of actions using the first portion of the object hierarchy at least in part by: using a first object of the first plurality of objects to cause, via a first active GUI element of the first plurality of active GUI elements, the first application program to at least partially perform the first action, the first action programmatically emulating a user's interaction with the first active GUI element; and after the first application program performs the first action, refreshing the object hierarchy to obtain a first refreshed object hierarchy comprising a third plurality of objects corresponding to a third plurality of active GUI elements reflecting changes in a user interface of the first application program; controlling the second application program to perform a second action in the sequence of actions using the second portion of the object hierarchy at least in part by: using a second object of the second plurality of objects to cause, via a second active GUI element of the second plurality of active GUI elements, the second application program to at least partially perform the second action, the second action programmatically emulating a user's interaction with the second active GUI element; and after the second application program performs the second action, refreshing the object hierarchy to obtain a second refreshed object hierarchy comprising a fourth plurality of objects corresponding to a fourth plurality of active GUI elements reflecting changes in a user interface of the second application program. ### Claim 25 (depends on 24) The at least one non-transitory computer-readable storage medium of claim 24, wherein refreshing the object hierarchy comprises refreshing only a subset of objects in the object hierarchy.