Back to results
US 10,310,701 B2Granted
Software robots for programmatically controlling computer programs to perform tasks
AssigneeSoroco Private Limited
InventorsGeorge Peter Nychis, Rohan Narayan Murty
Life of this patent
Estimating the term…
Abstract
A system comprising at least one computer hardware processor configured to perform: generating an object hierarchy comprising a plurality of objects corresponding to active graphical user interface (GUI) elements of at least one application program; controlling the at least one application program to perform a task comprising a sequence of actions at least in part by using the object hierarchy to invoke actions in the sequence of actions.
Claims25
A system for controlling at least one application program for performing a task comprising a sequence of actions, the at least one application program including a first application program and a second application program, the first and second application programs configured to execute at least in part by using different graphical user interface (GUI) application libraries, the system comprising:
- at least one computer hardware processor;
- and
- at least one non-transitory computer-readable storage medium storing processor-executable instructions that, when executed by the at least one computer hardware processor, cause the at least one computer hardware processor to perform:
- generating an object hierarchy comprising a plurality of objects corresponding to active GUI elements of the at least one application program, the generating comprising:
- identifying a first plurality of active GUI elements of the first application program and a second plurality of active GUI elements of the second application program, wherein the first application program is configured to execute by using a first GUI application library and the second application program is configured to execute by using a second GUI application library that is different than the first GUI application library;
- generating a first portion of the object hierarchy comprising a first plurality of objects corresponding to the first plurality of active GUI elements of the first application program and a second portion of the object hierarchy comprising a second plurality of objects corresponding to the second plurality of active GUI elements of the second application program,
- controlling the first application program to perform a first action in the sequence of actions using the first portion of the object hierarchy at least in part by:
- using a first object of the first plurality of objects to cause, via a first active GUI element of the first plurality of active GUI elements, the first application program to at least partially perform the first action, the first action programmatically emulating a user's interaction with the first active GUI element;
- and
- after the first application program performs the first action, refreshing the object hierarchy to obtain a first refreshed object hierarchy comprising a third plurality of objects corresponding to a third plurality of active GUI elements reflecting changes in a user interface of the first application program;
- and
- controlling the second application program to perform a second action in the sequence of actions using the second portion of the object hierarchy at least in part by:
- using a second object of the second plurality of objects to cause, via a second active GUI element of the second plurality of active GUI elements, the second application program to at least partially perform the second action, the second action programmatically emulating a user's interaction with the second active GUI element;
- and
- after the second application program performs the second action, refreshing the object hierarchy to obtain a second refreshed object hierarchy comprising a fourth plurality of objects corresponding to a fourth plurality of active GUI elements reflecting changes in a user interface of the second application program.
12 more elements
A method for controlling at least one application program for performing a task comprising a sequence of actions, the at least one application program including a first application program and a second application program, the first and second application programs configured to execute at least in part by using different graphical user interface (GUI) application libraries, the method comprising:
- using at least one computer hardware processor to perform:
- generating an object hierarchy comprising a plurality of objects corresponding to active GUI elements of the at least one application program, the generating comprising:
- identifying a first plurality of active GUI elements of the first application program and a second plurality of active GUI elements of the second application program, wherein the first application program is configured to execute by using a first GUI application library and the second application program is configured to execute by using a second GUI application library that is different than the first GUI application library;
- generating a first portion of the object hierarchy comprising a first plurality of objects corresponding to the first plurality of active GUI elements of the first application program and a second portion of the object hierarchy comprising a second plurality of objects corresponding to the second plurality of active GUI elements of the second application program;
- controlling the first application program to perform a first action in the sequence of actions using the first portion of the object hierarchy at least in part by:
- using a first object of the first plurality of objects to cause, via a first active GUI element of the first plurality of active GUI elements, the first application program to at least partially perform the first action, the first action programmatically emulating a user's interaction with the first active GUI element;
- and
- after the first application program performs the first action, refreshing the object hierarchy to obtain a first refreshed object hierarchy comprising a third plurality of objects corresponding to a third plurality of active GUI elements reflecting changes in a user interface of the first application program;
- and
- controlling the second application program to perform a second action in the sequence of actions using the second portion of the object hierarchy at least in part by:
- using a second object of the second plurality of objects to cause, via a second active GUI element of the second plurality of active GUI elements, the second application program to at least partially perform the second action, the second action programmatically emulating a user's interaction with the second active GUI element;
- and
- after the second application program performs the second action, refreshing the object hierarchy to obtain a second refreshed object hierarchy comprising a fourth plurality of objects corresponding to a fourth plurality of active GUI elements reflecting changes in a user interface of the second application program.
10 more elements
At least one non-transitory computer-readable storage medium storing processor-executable instructions that, when executed by at least one computer hardware processor, causes the at least one computer hardware processor to perform a method for controlling at least one application program for performing a task comprising a sequence of actions, the at least one application program including a first application program and a second application program, the first and second application programs configured to execute at least in part by using different graphical user interface (GUI) application libraries, the method comprising:
- generating an object hierarchy comprising a plurality of objects corresponding to active GUI elements of the at least one application program, the generating comprising:
- identifying a first plurality of active GUI elements of the first application program and a second plurality of active GUI elements of the second application program, wherein the first application program is configured to execute by using a first GUI application library and the second application program is configured to execute by using a second GUI application library that is different than the first GUI application library;
- generating a first portion of the object hierarchy comprising a first plurality of objects corresponding to the first plurality of active GUI elements of the first application program and a second portion of the object hierarchy comprising a second plurality of objects corresponding to the second plurality of active GUI elements of the second application program;
- controlling the first application program to perform a first action in the sequence of actions using the first portion of the object hierarchy at least in part by:
- using a first object of the first plurality of objects to cause, via a first active GUI element of the first plurality of active GUI elements, the first application program to at least partially perform the first action, the first action programmatically emulating a user's interaction with the first active GUI element;
- and
- after the first application program performs the first action, refreshing the object hierarchy to obtain a first refreshed object hierarchy comprising a third plurality of objects corresponding to a third plurality of active GUI elements reflecting changes in a user interface of the first application program;
- controlling the second application program to perform a second action in the sequence of actions using the second portion of the object hierarchy at least in part by:
- using a second object of the second plurality of objects to cause, via a second active GUI element of the second plurality of active GUI elements, the second application program to at least partially perform the second action, the second action programmatically emulating a user's interaction with the second active GUI element;
- and
- after the second application program performs the second action, refreshing the object hierarchy to obtain a second refreshed object hierarchy comprising a fourth plurality of objects corresponding to a fourth plurality of active GUI elements reflecting changes in a user interface of the second application program.
8 more elements
Dependencies are inferred from the claim text.