Enabling natural language interactions with user interfaces for users of a software application

A computer-implemented method for interacting with a user interface of a device using a natural language command, the computer-implemented method comprising:

• receiving a search request;
• displaying, within a graphical user interface of the device, a set of search results corresponding to the search request;
• receiving a natural language command from a user;
• identifying a set of user interface elements visible within the graphical user interface of the device;
• assembling user interface metadata describing the identified set of user interface elements;
• providing the language command and the user interface metadata to a natural language understanding (NLU) engine;
• receiving, from the NLU engine, a descriptor of a user interface element that is within the set of user interface elements and that is referenced by the natural language command, wherein the NLU engine uses a grammar corresponding to the device to determine the descriptor of the user interface element, the grammar being a context-free grammar having a slot and containing two numerical non-terminals indicating a row and a column of a selected one of the visible user interface elements;
• and
• using the descriptor to respond to the natural language command.

A multi-tenant system comprising:

• a computer processor;
• and
• a non-transitory computer-readable storage medium storing instructions that when executed by the computer processor perform actions comprising:
• receiving, from a device, a natural language command and user interface metadata describing a set of user interface elements visible within a user interface of the device;
• selecting a grammar corresponding to the device, the grammar being a context-free grammar and having a slot and containing two numerical non-terminals indicating a row and a column of a selected one of the visible user interface elements;
• parsing the natural language command according to the selected grammar, the slot being filled by text;
• matching the text with a user interface element in the user interface metadata;
• and
• providing a descriptor of the matched user interface element to the device.