Software robots for programmatically controlling computer programs to perform tasks

A system for remotely controlling one or more application programs executing on a respective one or more virtual machines, the system comprising:

• a controller, comprising a processor, configured to perform:
• accessing a software robot computer program for controlling at least one application program to perform a task including a first sub-task to be performed by a first application program executing on a first virtual machine;
• identifying, using the software robot computer program, a first action to perform in furtherance of the first sub-task;
• and
• providing an indication to the computing device to control the first application program to at least partially perform the first action;
• and
• a computing device, communicatively coupled to the controller, comprising a processor configured to execute the first virtual machine and to perform:
• generating a first object hierarchy comprising a first plurality of objects corresponding to active graphical user interface (GUI) elements of the first application program;
• and
• in response to receiving the indication from the controller device to at least partially perform the first action, using the first object hierarchy to cause the first application program to at least partially perform the first action.

A system for remotely controlling multiple application programs executing on multiple respective physical computing devices, the system comprising:

• a first computing device, comprising a first processor, and configured to execute a first application program;
• a second computing device, comprising a second processor, and configured to execute a second application program;
• and
• a controller communicatively coupled to the first and second computing devices and comprising a third processor, the controller configured to perform:
• generating, based on first information obtained from the first computing device and second information obtained from the second computing device, a global object hierarchy comprising a plurality of objects corresponding to active graphical user interface (GUI) elements of the first application program and the second application program;
• controlling the first application program to perform the first sub-task at least in part by using the global object hierarchy;
• and
• controlling the second application program to perform the second sub-task at least in part by using the global object hierarchy.

A system for remotely controlling one or more application programs executing on a respective one or more physical computing devices, the system comprising:

• a controller comprising a processor and configured to perform:
• accessing a software robot computer program for controlling at least one application program to perform a task including a first sub-task to be performed by a first application program executing on a first computing device external to the controller;
• identifying, using the software robot computer program, a first action to perform in furtherance of the first sub-task;
• and
• providing an indication to the first computing device to control the first application program to at least partially perform the first action;
• and
• the first computing device, communicatively coupled to the controller and comprising a processor, configured to execute the first application program and to perform:
• generating a first object hierarchy comprising a first plurality of objects corresponding to active graphical user interface (GUI) elements of the first application program;
• and
• in response to receiving the indication from the controller to at least partially perform the first action, using the first object hierarchy to cause the first application program to at least partially perform the first action.