Framework for UI automation based on graph recognition technology and related methods

A computing system comprising:

• a graphical user interface (GUI) testing device in communication with a computing device configured to execute a software application with an associated GUI, the GUI testing device configured to
• execute a testing state machine for interacting with the software application to generate an initial screen of the GUI, the testing state machine comprising a plurality of states,
• determine a current state in the testing state machine based upon a matching trigger target in the initial screen to a given state, the current state comprising at least one operation, and at least one trigger associated with the at least one operation, the at least one trigger including a source state, a destination state, and a trigger target, the at least one operation comprising a user input operation, and an operation trigger target,
• the determining of the current state comprising
• applying a convolutional neural network (CNN) to generate a plurality of labels, and
• finding the trigger including the matching trigger target in the plurality of labels, the matching trigger including a source state being the current state, and
• perform the at least one operation on the matching trigger target in the initial screen to generate a next screen of the GUI, and advance from the current state to a next state based upon the at least one trigger.