Software robots for programmatically controlling computer programs to perform tasks

A method for controlling at least one application program to perform a task comprising a first sub-task to be performed by a first application program, the method comprising:

• using at least one computer hardware processor to perform:
• generating an object hierarchy comprising a plurality of objects corresponding to active graphical user interface (GUI) elements of the first application program, the generating comprising:
• identifying a first plurality of active GUI elements associated with a user interface of the first application program;
• and
• generating, as part of the object hierarchy, a first plurality of objects corresponding to the first plurality of active GUI elements;
• and
• controlling the first application program to perform the first sub-task at least in part by:
• identifying, using a computer program, a first action to perform in furtherance of the first sub-task;
• accessing, in the object hierarchy, a first object of the first plurality of objects corresponding to a first active GUI element of the first plurality of active GUI elements of the first application program;
• using the first object to cause, via the first active GUI element, the first application program to at least partially perform the first action;
• identifying, using the computer program, a second action to perform in furtherance of the first sub-task;
• refreshing the object hierarchy to obtain a refreshed object hierarchy comprising a second plurality of objects corresponding to a second plurality of active GUI elements associated with the user interface of the first application program, the second plurality of objects including a second object not in the first plurality of objects, the second object corresponding to a second active GUI element of the second plurality of active GUI elements;
• and
• using the second object to cause, via the second active GUI element, the first application program to at least partially perform the second action.

A non-transitory computer-readable storage medium storing processor-executable instructions that, when executed by at least one hardware processor, cause the at least one hardware processor to perform a method for controlling at least one application program to perform a task comprising a first sub-task to be performed by a first application program, the method comprising:

• generating an object hierarchy comprising a plurality of objects corresponding to active graphical user interface (GUI) elements of the first application program, the generating comprising:
• identifying a first plurality of active GUI elements associated with a user interface of the first application program;
• and
• generating, as part of the object hierarchy, a first plurality of objects corresponding to the first plurality of active GUI elements;
• and
• controlling the first application program to perform the first sub-task at least in part by:
• identifying, using a computer program, a first action to perform in furtherance of the first sub-task;
• accessing, in the object hierarchy, a first object of the first plurality of objects corresponding to a first active GUI element of the first plurality of active GUI elements of the first application program;
• using the first object to cause, via the first active GUI element, the first application program to at least partially perform the first action;
• identifying, using the computer program, a second action to perform in furtherance of the first sub-task;
• refreshing the object hierarchy to obtain a refreshed object hierarchy comprising a second plurality of objects corresponding to a second plurality of active GUI elements associated with the user interface of the first application program, the second plurality of objects including a second object not in the first plurality of objects, the second object corresponding to a second active GUI element of the second plurality of active GUI elements;
• and
• using the second object to cause, via the second active GUI element, the first application program to at least partially perform the second action.