Using natural language to control structured web page data

A system for multi-modal interaction with structured web page data, the system comprising:

• a memory having computer-readable instructions;
• and
• one or more processors for executing the computer-readable instructions, the computer-readable instructions comprising:
• instructions for automatically injecting event monitoring code into a web page, the injected event monitoring code comprises identifiers for one or more display elements that include content to be displayed on the web page, and for one or more original executable code elements that include computer programs to operate on the content;
• instructions for receiving one or more event streams generated by the injected event monitoring code, an event stream comprising one or more semantic mappings between the identifiers for the one or more original executable code elements and the one or more display elements;
• instructions for receiving from a user, a voice command that includes a request to perform an operation to create new content using the content already displayed in the one or more display elements of the web page, the operation not being in the original executable code elements;
• instructions for generating a computer program having the one or more content elements as parameters and adding semantic mappings between the computer program to be executed as a command responsive to the received request;
• adding, using the processor, the command as a new executable code element of the web page;
• and
• instructions for executing the command to fulfil the received request by generating new content that is displayed on the web page, wherein generating the new content comprises adding one or more new structured display elements to the web page, the one or more new structured display elements comprising the new content that is dynamically generated based on the content that was already displayed on the web page.

A computer-program product for multi-modal interaction with structured web page data, the computer-program product comprising a computer-readable storage memory device having program instructions embodied therewith, the program instructions executable by a processor to cause the processor to perform a method comprising:

• automatically injecting event monitoring code into a web page, the injected event monitoring code comprises identifiers for one or more display elements that include content to be displayed on the web page, and for one or more original executable code elements that include computer programs to operate on the content;
• receiving one or more event streams generated by the injected event monitoring code, an event stream comprising one or more semantic mappings between the identifiers for the one or more original executable code elements and the one or more display elements;
• receiving from a user, a voice command that includes a request to perform an operation to create new content using the content displayed in the one or more content elements of the web page, the operation not being in the original executable code elements;
• generating a computer program having the one or more content elements as parameters and adding semantic mappings between the computer program to be executed as a command responsive to the received request;
• adding, using the processor, the command as a new executable code element of the web page;
• and
• executing the command to fulfil the received request by generating the new content that is displayed on the web page, wherein generating the new content comprises adding one or more new structured display elements to the web page, the one or more new structured content elements comprising the new content that is dynamically generated based on the content that was already displayed on the web page.