Automated remedial action to expose a missing target and/or anchor(s) for user interface automation

A computer-implemented method for performing target graphical element identification in a user interface (UI) automation, comprising:

• searching, by a robotic process automation (RPA) robot, for a target graphical element and one or more visible anchors matching anchors associated with the target graphical element;
• and
• when the target graphical element is not found in the search, attempting a remedial action to make the target graphical element visible, by the RPA robot, wherein
• when the remedial action is not successful in allowing the RPA robot to find the target graphical element, the method further comprises:
• receiving a bounding box or an indication identifying the undetected target graphical element in the UI, by the RPA robot;
• determining a UI descriptor for the target graphical element associated with the bounding box or the indication, by the RPA robot;
• and
• updating logic associated with the UI automation or retraining a machine learning (ML) model to use the determined UI descriptor.

A non-transitory computer-readable medium storing a computer program for performing target graphical element detection in a user interface (UI) automation, the computer program configured to cause at least one processor to:

• search for a target graphical element and one or more visible anchors matching anchors associated with the target graphical element;
• and
• when the target graphical element is not found in the search:
• receive information from a server running a virtual machine (VM) pertaining to applications that are running, windows that are open, and locations of the open windows, and
• attempt a remedial action by using the received information to generate mouse click information, key press information, or both, that cause the server to move a window associated with the target graphical element into a foreground of the UI, launch an application associated with the target graphical element, or both, wherein
• when the remedial action is not successful in allowing the UI automation to find the target graphical element, the computer program is further configured to cause the at least one processor to:
• start a recorder that observes remedial user interactions with a computing system,
• record the remedial user interactions, and
• use the recorded remedial user interactions to update logic associated with the UI automation or retrain a machine learning (ML) model to find the target graphical element.