Enhanced visual dialog system for intelligent tutors

A computer program product, comprising:

• a computer-readable storage medium having computer readable program code embodied therewith, the computer readable program code executable by a processor to perform an operation comprising:
• receiving natural language text input via a chat interface of a tutor application;
• identifying, using at least one classifier applied to the text input, a concept in the text input;
• mapping the concept in the text input to a first visual object currently displayed on a graphical user interface (GUI);
• upon mapping the concept in the text input to the first visual object, generating, based on a first machine learning (ML) model, a first program code statement corresponding to the first visual object, wherein the first program code statement is code in a machine-executable programming language;
• executing the first program code statement to modify a visualization interface of the tutor application based on the text input received via the chat interface by changing depiction of the first visual object on the GUI;
• receiving input requesting to modify the first visual object;
• generating a second program code statement to modify the first visual object based on the input;
• generating, by processing the second program code statement using a second ML model, a natural language statement describing the requested modification to the first visual object;
• and
• outputting the natural language statement via the chat interface.

A system, comprising:

• a processor;
• and
• a memory storing one or more instructions which, when executed by the processor, performs an operation comprising:
• receiving natural language text input via a chat interface of a tutor application;
• identifying, using at least one classifier applied to the text input, a concept in the text input;
• mapping the concept in the text input to a first visual object currently displayed on a graphical user interface (GUI);
• upon mapping the concept in the text input to the first visual object, generating, based on a first machine learning (ML) model, a first program code statement corresponding to the first visual object, wherein the first program code statement is code in a machine-executable programming language;
• and
• executing the first program code statement to modify a visualization interface of the tutor application based on the text input received via the chat interface by changing depiction of the first visual object on the GUI;
• receiving input requesting to modify the first visual object;
• generating a second program code statement to modify the first visual object based on the input;
• generating, by processing the second program code statement using a second ML model, a natural language statement describing the requested modification to the first visual object;
• and
• outputting the natural language statement via the chat interface.