Agent traversal of multi-layer data structure

One or more non-transitory computer-readable media comprising instructions that, upon execution of the instructions by one or more processors of a computing device, are to cause the computing device to:

• identify a natural language query received from a user at an initial state of a user interface (UI) of an electronic device, wherein the natural language query relates to a target task performable at a terminal state of the UI;
• identify a multi-layer data structure related to a plurality of states of the UI, wherein the plurality of states of the UI include at least the initial state of the UI, the terminal state of the UI, and one or more intermediate states of the UI that are between the initial state of the UI and the terminal state of the UI;
• process, via a multi-modal machine-learning model, the natural language query and the multi-layer data structure to identify the terminal state of the UI at which the target task is performable;
• and
• cause, based on the identified terminal state of the UI, the target task to be performed.

An electronic system comprising:

• memory to store a natural language query received from a user at an initial state of a user interface (UI) of an electronic device, wherein the natural language query relates to a target task performable at a terminal state of the UI;
• and
• one or more processors configured to:
• identify a multi-layer data structure related to a plurality of states of the UI, wherein the plurality of states of the UI include at least the initial state of the UI, the terminal state of the UI, and one or more intermediate states of the UI that are between the initial state of the UI and the terminal state of the UI;
• process, via a multi-modal machine-learning model, the natural language query and the multi-layer data structure to identify the terminal state of the UI at which the target task is performable;
• and
• cause, based on the identified terminal state of the UI, the target task to be performed.