Automatic navigation of interactive web documents

A method implemented using one or more processors, comprising:

• determining a set of keys that are applicable to an interactive web document that is operable via a graphical user interface (“GUI”) to perform a task, wherein the interactive web document includes a plurality of constituent interactive document object model (DOM) elements that are operable to input a corresponding plurality of values;
• automatically populating at least some of the plurality of constituent interactive DOM elements with arbitrary values to generate a target state of the interactive web document;
• encoding at least the populated interactive DOM elements of the plurality of interactive elements into a plurality of interactive DOM element feature vectors;
• selecting a key from the set of keys;
• encoding at least the selected key into a key feature vector;
• based on the key feature vector and the plurality of interactive DOM element feature vectors, calculating one or more probability distributions over the populated interactive DOM elements;
• selecting an interactive DOM element from the populated interactive DOM elements based on the one or more probability distributions;
• and
• generating an instruction based on the selected key and the arbitrary value used to populate the selected interactive DOM element.

A system comprising one or more processors and memory storing instructions that, in response to execution of the instructions by the one or more processors, cause the one or more processors to:

• determine a natural language input, wherein the natural language input comprises a command to perform a task;
• analyze the natural language input to identify one or more key-value pairs;
• retrieve an interactive web document that is operable via a graphical user interface (“GUI”) to perform the task, wherein the interactive web document includes one or more constituent document object model (DOM) interactive elements that are operable to input one or more values of the one or more key-value pairs;
• encode the one or more key-value pairs into one or more instruction feature vectors;
• encode overlapping content between the one or more key-value pairs and the one or more interactive DOM elements into one or more overlap feature vectors;
• encode the one or more interactive DOM elements of the interactive web document into one or more interactive DOM element feature vectors;
• condition the one or more interactive DOM element feature vectors based on the one or more overlap feature vectors to generate one or more conditioned interactive DOM element feature vectors, wherein the instructions to condition include instructions to perform an operation using the one or more interactive DOM element feature vectors and the one or more overlap feature vectors as operands;
• generate, based on the one or more conditioned interactive element feature vectors and the one or more instruction feature vectors, one or more probability distributions over the one or more interactive elements;
• and
• facilitate automated navigation through the interactive web document in response to the natural language input based at least in part on one or more of the probability distributions.