# US 11,307,876 B1 · Automated remedial action to expose a missing target and/or anchor(s) for user interface automation > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,307,876 B1 - Document ID: US11307876B1 - Status: Granted - Kind: B1 - Application number: 17/101,036 - Filed: 2020-11-23 - Granted: 2022-04-19 - Earliest claimed: 2020-11-20 - Assignees: UiPath, Inc. - Applicants: UiPath, Inc. - Inventors: Michael Aristo Leonard, II; Cosmin Voicu - Web page: https://patentgrep.com/patent/US11307876B1 ## CPC - G06F 9/451: Execution arrangements for user interfaces - G06F 3/04812: Interaction techniques based on cursor appearance or behaviour, e.g. being affected by the presence of displayed objects - G06F 3/04817: using icons (graphical or visual programming using iconic symbols G06F8/34) - G06F 3/0482: Interaction with lists of selectable items, e.g. menus - G06F 3/04845: for image manipulation, e.g. dragging, rotation, expansion or change of colour - G06F 11/302: {where the computing system component is a software system} - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract Automatic anchor determination for target graphical element identification in user interface (UI) automation is disclosed. A context-based mechanism assists in discriminating between duplicate target UI element candidates. More specifically, additional anchors may be determined and automatically added for a target UI element that provide context and are visible in an area surrounding the target. During design time, a target UI element may be indicated by a user of a designer application and a corresponding anchor may be determined. When a pair of UI elements is found having the same or similar characteristics and/or relationships to the target-anchor pair, an additional anchor is automatically identified without requesting user input. The additional anchor may be selected from the UI elements within a radius of the target UI element. ## Claims ### Claim 1 A computing system, comprising: memory storing computer program instructions for performing target graphical element detection in a user interface (UI) automation; and at least one processor configured to execute the computer program instructions, wherein the computer program instructions are configured to cause the at least one processor to: search for a target graphical element and one or more visible anchors matching anchors associated with the target graphical element, when the target graphical element is found in the search, use geometric relationships between the one or more matching visible anchors and the target graphical element to identify the target graphical element in a UI, and when the target graphical element is not found in the search, attempt a remedial action to make the target graphical element visible, wherein the remedial action comprises moving a mouse pointer to a location in the UI associated with a detected anchor or a location adjacent to the detected anchor, simultaneously simulating presses of ALT and SPACE keys or CTRL and ALT and SPACE keys, choosing a “Move” menu option, and moving an application window associated with the target graphical element to be more visible or entirely visible. ### Claim 2 (depends on 1) The computing system of claim 1, wherein the remedial action comprises simultaneously simulating presses of ALT and TAB keys and selecting an application window associated with the target graphical element to bring the application window to a front of the UI. ### Claim 3 (depends on 1) The computing system of claim 1, wherein the remedial action comprises moving one or more other windows that are blocking an application window associated with the target graphical element. ### Claim 4 (depends on 1) The computing system of claim 1, wherein the remedial action comprises attempting to open an application associated with the target graphical element when the application is not already open. ### Claim 5 (depends on 1) The computing system of claim 1, wherein the remedial action comprises using operating system-level application programming interface (API) calls to make an application window associated with the target graphical element visible or to open an application associated with the target graphical element when the application is not already open. ### Claim 6 (depends on 1) The computing system of claim 1, wherein the computer program instructions are configured to cause the at least one processor to: wait for a period of time during which a user has not interacted with the computing system before attempting the remedial action. ### Claim 7 (depends on 1) The computing system of claim 1, wherein the computer program instructions are configured to cause the at least one processor to: wait for a time when a user is unlikely to interact with the computing system before attempting the remedial action. ### Claim 8 (depends on 1) The computing system of claim 1, wherein the UI automation is executed in a virtual machine (VM) environment and the remedial action comprises: clicking a task bar icon for an application associated with the target graphical element to launch the application. ### Claim 9 (depends on 1) The computing system of claim 1, wherein the UI automation is executed in a virtual machine (VM) environment and the remedial action comprises: receiving information from a server running the VM pertaining to applications that are running, windows that are open, and locations of the open windows; and using the received information to generate mouse click information, key press information, or both, that cause the server to move a window associated with the target graphical element into a foreground of the UI, launch an application associated with the target graphical element, or both. ### Claim 10 (depends on 1) The computing system of claim 1, wherein when the remedial action is not successful in allowing the UI automation to find the target graphical element, the computer program instructions are configured to cause the at least one processor to: receive a bounding box or an indication identifying the undetected target graphical element in the UI; determine a UI descriptor for the target graphical element associated with the bounding box or the indication; and update logic associated with the UI automation or retrain a machine learning (ML) model to use the determined UI descriptor. ### Claim 11 (depends on 1) The computing system of claim 1, wherein when the remedial action is not successful in allowing the UI automation to find the target graphical element, the computer program instructions are configured to cause the at least one processor to: start a recorder that observes remedial user interactions with the computing system; record the remedial user interactions; and use the recorded remedial user interactions to update logic associated with the UI automation or retrain a machine learning (ML) model to find the target graphical element. ### Claim 12 (depends on 11) The computing system of claim 11, wherein the UI automation is performed by a robotic process automation (RPA) robot. ### Claim 13 A computer-implemented method for performing target graphical element identification in a user interface (UI) automation, comprising: searching, by a robotic process automation (RPA) robot, for a target graphical element and one or more visible anchors matching anchors associated with the target graphical element; and when the target graphical element is not found in the search, attempting a remedial action to make the target graphical element visible, by the RPA robot, wherein when the remedial action is not successful in allowing the RPA robot to find the target graphical element, the method further comprises: receiving a bounding box or an indication identifying the undetected target graphical element in the UI, by the RPA robot; determining a UI descriptor for the target graphical element associated with the bounding box or the indication, by the RPA robot; and updating logic associated with the UI automation or retraining a machine learning (ML) model to use the determined UI descriptor. ### Claim 14 (depends on 13) The computer-implemented method of claim 13, wherein the remedial action comprises: simultaneously simulating presses of ALT and TAB keys and selecting an application window associated with the target graphical element to bring the application window to a front of the UI; moving a mouse pointer to a location in the UI associated with a detected anchor or a location adjacent to the detected anchor, simultaneously simulating presses of ALT and SPACE keys or CTRL and ALT and SPACE keys, choosing a “Move” menu option, and moving the application window associated with the target graphical element to be more visible or entirely visible; moving one or more other windows that are blocking the application window associated with the target graphical element; attempting to open an application associated with the target graphical element when the application is not already open; using operating system-level application programming interface (API) calls to make the application window associated with the target graphical element visible or to open the application associated with the target graphical element when the application is not already open; or a combination thereof. ### Claim 15 (depends on 13) The computer-implemented method of claim 13, further comprising: waiting for a period of time during which a user has not interacted with a computing system before attempting the remedial action, by the RPA robot. ### Claim 16 (depends on 13) The computer-implemented method of claim 13, wherein the UI automation is executed in a virtual machine (VM) environment and the remedial action comprises: clicking a task bar icon for an application associated with the target graphical element to launch the application. ### Claim 17 (depends on 13) The computer-implemented method of claim 13, wherein the UI automation is executed in a virtual machine (VM) environment and the remedial action comprises: receiving information from a server running the VM pertaining to applications that are running, windows that are open, and locations of the open windows; and using the received information to generate mouse click information, key press information, or both, that cause the server to move a window associated with the target graphical element into a foreground of the UI, launch an application associated with the target graphical element, or both. ### Claim 18 (depends on 13) The computer-implemented method of claim 13, wherein when the remedial action is not successful in allowing the RPA robot to find the target graphical element, the method further comprises: starting a recorder that observes remedial user interactions with a computing system, by the RPA robot; recording the remedial user interactions, by the RPA robot; and using the recorded remedial user interactions to update logic associated with the UI automation or retrain the ML model to find the target graphical element. ### Claim 19 A non-transitory computer-readable medium storing a computer program for performing target graphical element detection in a user interface (UI) automation, the computer program configured to cause at least one processor to: search for a target graphical element and one or more visible anchors matching anchors associated with the target graphical element; and when the target graphical element is not found in the search: receive information from a server running a virtual machine (VM) pertaining to applications that are running, windows that are open, and locations of the open windows, and attempt a remedial action by using the received information to generate mouse click information, key press information, or both, that cause the server to move a window associated with the target graphical element into a foreground of the UI, launch an application associated with the target graphical element, or both, wherein when the remedial action is not successful in allowing the UI automation to find the target graphical element, the computer program is further configured to cause the at least one processor to: start a recorder that observes remedial user interactions with a computing system, record the remedial user interactions, and use the recorded remedial user interactions to update logic associated with the UI automation or retrain a machine learning (ML) model to find the target graphical element. ### Claim 20 (depends on 19) The non-transitory computer-readable medium of claim 19, wherein when the remedial action is not successful in allowing the UI automation to find the target graphical element, the computer program is further configured to cause the at least one processor to: receive a bounding box or an indication identifying the undetected target graphical element in the UI; determine a UI descriptor for the target graphical element associated with the bounding box or the indication; and update logic associated with the UI automation or retraining the ML model to use the determined UI descriptor.