Training a system to perform a task with multiple specific steps given a general natural language command

A system for performing a task with multiple specific steps given a general natural language command, the system comprising:

• an electronic processor, the electronic processor configured to receive a general natural language command specifying a webpage and a task to perform on a graphical user interface of the webpage;
• using a first machine learning system, generate a plurality of specific steps associated with the general natural language command, wherein each specific step of the plurality of specific steps comprises an action to be performed on an element of the graphical user interface of the webpage;
• perform the task using the plurality of specific steps and a second machine learning system;
• determine whether the task is performed successfully;
• and
• when the task is not performed successfully:
• retrain the first machine learning system by recording user actions of a user performing the task on the webpage provided by the general natural language command;
• and/or
• retrain the second machine learning system by displaying a natural language description of a graphical user interface element to a user;
• recording the user selection of a graphical user element of the webpage;
• and associating the natural language description of the graphical user interface element with the user selection of the graphical user element of the webpage.