Analyzing graphical user interfaces to facilitate automatic interaction

A system comprising one or more processors and memory storing instructions that, in response to execution by the one or more processors, cause the one or more processors to:

• identify a target visual cue to be located in a graphical user interface (“GUI”) comprising an interactive webpage accessible at a uniform resource locator (“URL”);
• obtain a document object model (“DOM”) of the interactive webpage, wherein the DOM of the interactive webpage comprises one or more interactive elements;
• obtain a bitmap screenshot of the GUI;
• apply features of the bitmap screenshot and the DOM as inputs across a machine learning model to generate output;
• based on the output, identify one or more of the interactive elements of the GUI as corresponding to the target visual cue;
• and
• automatically populating the one or more identified interactive elements with data.

At least one non-transitory computer-readable medium comprising instructions that, in response to execution by one or more processors, cause the one or more processors to:

• identify a target visual cue to be located in a graphical user interface (“GUI”) comprising an interactive webpage accessible at a uniform resource locator (“URL”);
• obtain a document object model (“DOM”) of the interactive webpage, wherein the DOM of the interactive webpage comprises one or more interactive elements;
• obtain a bitmap screenshot of the GUI;
• apply features of the bitmap screenshot and the DOM as inputs across a machine learning model to generate output;
• based on the output, identify one or more of the interactive elements of the GUI as corresponding to the target visual cue;
• and
• automatically populating the one or more identified interactive elements with data.