Software robots for programmatically controlling computer programs to perform tasks

A method for controlling multiple application programs to perform a task, the method comprising:

• using at least one computer hardware processor to perform:
• generating an object hierarchy comprising a plurality of objects corresponding to graphical user interface (GUI) elements of the first and second application programs at least in part by:
• obtaining a first representation of GUI elements of the first application program from a first source,
• generating, based on the first representation of GUI elements of the first application program, a first portion of the object hierarchy corresponding to GUI elements of the first application program, the first portion of the object hierarchy comprising a first object corresponding to a first GUI element of the first application program,
• obtaining a second representation of GUI elements of the second application program from a second source different from the first source, and
• generating, based on the second representation of GUI elements of the second application program, a second portion of the object hierarchy corresponding to the GUI elements of the second application program, the second portion of the object hierarchy comprising a second object corresponding to a second GUI element of the second application program;
• controlling the first application program using the object hierarchy at least in part by:
• using the first object corresponding to the first GUI element of the first application program to cause, via the first GUI element, the first application program to at least partially perform a first action;
• and
• controlling the second application program using the object hierarchy at least in part by:
• using the second object corresponding to the second GUI element of the second application program to cause, via the second GUI element, the second application program to at least partially perform a second action.

At least one non-transitory computer-readable storage medium storing processor-executable instructions that, when executed by at least one hardware processor, cause the at least one hardware processor to perform a method for controlling multiple application programs to perform a task, the method comprising:

• generating an object hierarchy comprising a plurality of objects corresponding to graphical user interface (GUI) elements of the first and second application programs at least in part by:
• obtaining a first representation of GUI elements of the first application program from a first source,
• generating, based on the first representation of GUI elements of the first application program, a first portion of the object hierarchy corresponding to GUI elements of the first application program, the first portion of the object hierarchy comprising a first object corresponding to a first GUI element of the first application program,
• obtaining a second representation of GUI elements of the second application program from a second source different from the first source, and
• generating, based on the second representation of GUI elements of the second application program, a second portion of the object hierarchy corresponding to the GUI elements of the second application program, the second portion of the object hierarchy comprising a second object corresponding to a second GUI element of the second application program;
• controlling the first application program using the generated object hierarchy at least in part by:
• using the first object corresponding to the first GUI element of the first application program to cause, via the first GUI element, the first application program to at least partially perform a first action;
• and
• controlling the second application program using the object hierarchy at least in part by:
• using the second object corresponding to the second GUI element of the second application program to cause, via the second GUI element, the second application program to at least partially perform a second action.