Automatic detection of user interface elements

A system comprising:

• one or more computers;
• and
• a computer-readable medium coupled to the one or more computers having instructions stored thereon which, when executed by the one or more computers, cause the one or more computers to perform operations comprising:
• accessing master frame information for a user interface of an application, the master frame information including first captured user interface information captured during a first execution of a scenario for the application;
• performing a subsequent execution of the scenario, including capturing second captured user interface information;
• determining that the subsequent execution of the scenario has not passed successfully;
• in response to determining that the subsequent execution of the scenario has not passed successfully, determining that the subsequent execution of the scenario has failed due to a non-functional error;
• determining that the non-functional error is based on a failure to locate a user interface element specified in the master frame;
• and
• performing a recovery strategy, using the first captured user interface information and the second captured user interface information, to automatically locate the user interface element.

A computer program product encoded on a non-transitory storage medium, the product comprising non-transitory, computer readable instructions for causing one or more processors to perform operations comprising:

• accessing master frame information for a user interface of an application, the master frame information including first captured user interface information captured during a first execution of a scenario for the application;
• performing a subsequent execution of the scenario, including capturing second captured user interface information;
• determining that the subsequent execution of the scenario has not passed successfully;
• in response to determining that the subsequent execution of the scenario has not passed successfully, determining that the subsequent execution of the scenario has failed due to a non-functional error;
• determining that the non-functional error is based on a failure to locate a user interface element specified in the master frame;
• and
• performing a recovery strategy, using the first captured user interface information and the second captured user interface information, to automatically locate the user interface element.