Visual programming methods and systems for intent dispatch

A non-transitory computer readable medium containing program instructions that when executed, cause a computer to:

• receive, from a user via a input device, a command phrase;
• process the command phrase to determine an entity name;
• identify a graphical user interface control corresponding to the entity name by analyzing a screen capture of a graphical user interface using a trained convolutional neural network;
• and
• responsive to identifying the graphical user interface control, display, on the graphical user interface, an action palette relating to a palette intent wherein the action palette includes a plurality of different action types.

A computer-implemented method for creating custom voice actions in an enterprise computing environment using an action palette, comprising:

• receiving, by one or more processors from a user via an input device, a command phrase;
• processing, by the one or more processors, the command phrase to determine an entity name;
• identifying, by the one or more processors, a graphical user interface control corresponding to the entity name by analyzing a screen capture of a graphical user interface using a trained convolutional neural network;
• and
• responsive to identifying the graphical user interface control, displaying, by the one or more processors on the graphical user interface, the action palette relating to a palette intent wherein the action palette includes a plurality of different action types.

A computing system, comprising:

• one or more processors;
• an input device;
• and
• a memory containing instructions that, when executed, cause the computing system to:
• receive, from a user via the input device, user input specifying a graphical user interface control, wherein the user input comprises (i) a textual label, or (ii) a control type;
• provide the user input and a screen capture of a graphical user interface to a trained convolutional neural network to cause the trained convolutional neural network to (i) analyze the screen capture to identify the graphical user interface control specified by the user input, and (ii) return an identifier for the graphical user interface control;
• and
• responsive to receiving the identifier for the graphical user interface control, display, on the graphical user interface, an action palette comprising a plurality of different action types.