Systems and methods of multimodal user interface automation

A device comprising:

• a communication interface;
• and
• one or more processors configured to:
• store in a memory an objective for automated manipulation of a user interface (UI) of a user device;
• conduct an iterative sequence to achieve the objective, wherein the iterative sequence includes:
• capturing current state information of the UI, wherein the current state information includes text and an image of the UI,
• detecting changes in the current state information from stored previous state information,
• selecting, from a component hierarchy of code for the UI, relevant sections related to the changes in the current state information,
• sending multimodal input to a generative artificial intelligence (AI) system, wherein the multimodal input includes the image of the UI, a text prompt, the objective, and the relevant sections,
• receiving, from the generative AI system, a predicted next action for the UI based on the multimodal input,
• initiating a programmatic action to implement the predicted next action on the UI, and
• recording the programmatic action;
• and
• generating, based on the iterative sequence, a UI automation model associated with achieving the objective.

A non-transitory, computer-readable storage medium storing instructions executable by a processor of a computing device for:

• storing, in a memory, an objective for automated manipulation of a user interface (UI) of a user device;
• conducting, by the computing device, an iterative sequence to achieve the objective, wherein the iterative sequence includes:
• capturing current state information of the UI, wherein the current state information includes text and an image of the UI,
• detecting changes in the current state information from stored previous state information,
• selecting, from a component hierarchy of code for the UI, relevant sections related to the changes in the current state information,
• sending multimodal input to a generative artificial intelligence (AI) system, wherein the multimodal input includes the image of the UI, a text prompt, the objective, and the relevant sections,
• receiving, from the generative AI system, a predicted next action for the UI based on the multimodal input,
• initiating a programmatic action to implement the predicted next action on the UI, and
• recording the programmatic action;
• and
• generating, by the computing device and based on the iterative sequence, a UI automation model associated with achieving the objective.