# US 11,442,991 B2 · Using natural language to control structured web page data > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,442,991 B2 - Document ID: US11442991B2 - Status: Granted - Kind: B2 - Application number: 16/520,929 - Filed: 2019-07-24 - Published: 2020-03-26 - Granted: 2022-09-13 - Earliest claimed: 2018-09-21 - Assignees: International Business Machines Corporation; Rensselaer Polytechnic Institute - Applicants: International Business Machines Corporation; Rensselaer Polytechnic Institute - Inventors: Jeffrey Owen Kephart; Hui Su; Matthew Peveler - Web page: https://patentgrep.com/patent/US11442991B2 ## CPC - G06F 16/90332: {Natural language query formulation or dialogue systems} - G06F 3/167: {Audio in a user interface, e.g. using voice commands for navigating, audio feedback} - G06F 16/9532: Query formulation - G06F 16/954: Navigation, e.g. using categorised browsing - G06F 40/30: Semantic analysis - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract A computer-implemented method for multi-modal interaction with structured web page data includes automatically injecting event monitoring code into a web page. One or more event streams generated by the injected event monitoring code are received. A request to perform an operation on data displayed on the web page is received from a user. A command is generated that has one or more parameters based at least in part on one or more mappings between the received request and the received one or more event streams. The command to fulfil the received request is executed by interacting with the web page. ## Claims ### Claim 1 A computer-implemented method for multi-modal interaction with structured web page data, the computer-implemented method comprising: automatically injecting, using a processor, event monitoring code into a web page, the injected event monitoring code comprises identifiers for one or more display elements that include content to be displayed on the web page, and for one or more original executable code elements that include computer programs to operate on the content; receiving, using the processor, one or more event streams generated by the injected event monitoring code, an event stream comprising one or more semantic mappings between the identifiers for the one or more original executable code elements and the one or more display elements; receiving, using the processor, from a user, a voice command that includes a request to perform an operation to create new content using the content already displayed in the one or more display elements of the web page, the operation not being in the original executable code elements; generating, using the processor, a computer program having the one or more content elements as parameters and adding semantic mappings between the computer program to be executed as a command responsive to the received request; adding, using the processor, the command as a new executable code element of the web page; and executing, using the processor, the command to fulfil the received request by generating the new content that is displayed on the web page, wherein generating the new content comprises adding one or more new structured display elements to the web page, the one or more new structured display elements comprising the new content that is dynamically generated based on the content that is already displayed on the web page. ### Claim 2 (depends on 1) The computer-implemented method of claim 1, wherein the received request comprises voice input. ### Claim 3 (depends on 1) The computer-implemented method of claim 1, wherein the one or more new structured content elements comprise new columns added to a table that is already displayed on the web page. ### Claim 4 (depends on 1) The computer-implemented method of claim 1, wherein the one or more mappings comprise domain-specific semantic mappings. ### Claim 5 (depends on 1) The computer-implemented method of claim 1, wherein the injected event monitoring code is associated with specific Document Object Model (DOM) elements within the web-page. ### Claim 6 (depends on 5) The computer-implemented method of claim 5, wherein generating the command further comprises: determining if at least one of the one or more parameters is invalid based on the text elements included in the received request; retrieving one or more events extracted from the one or more event streams; mapping the text elements to at least one of the one or more received events associated with the DOM elements based on semantic linkages between the text elements and the DOM elements: and inferring a value of the at least one invalid parameter based on the mapping. ### Claim 7 (depends on 6) The computer-implemented method of claim 6, wherein inferring the value further comprises performing semantical analysis on the web page document to identify DOM elements of interest, the DOM elements of interest producing the mapping. ### Claim 8 (depends on 3) The computer-implemented method of claim 3, wherein rendering the results further comprises automatically injecting code into the web page to render a modified web page. ### Claim 9 (depends on 3) The computer-implemented method of claim 3, wherein rendering the results further comprises rendering the results using a new web page. ### Claim 10 (depends on 3) The computer-implemented method of claim 3, wherein rendering the results further comprises rendering the results using synthesized speech. ### Claim 11 (depends on 1) The computer-implemented method of claim 1, wherein the request to perform an operation on data displayed on the web page further comprises the request to perform a statistical operation over the data displayed on the web page. ### Claim 12 (depends on 1) The computer-implemented method of claim 1, wherein the one or more parameters are inferred from one or more user-supplied domain-specific semantic mappings between text elements extracted from the received request and one or more labels associated with one or more events extracted from the one or more event streams. ### Claim 13 A system for multi-modal interaction with structured web page data, the system comprising: a memory having computer-readable instructions; and one or more processors for executing the computer-readable instructions, the computer-readable instructions comprising: instructions for automatically injecting event monitoring code into a web page, the injected event monitoring code comprises identifiers for one or more display elements that include content to be displayed on the web page, and for one or more original executable code elements that include computer programs to operate on the content; instructions for receiving one or more event streams generated by the injected event monitoring code, an event stream comprising one or more semantic mappings between the identifiers for the one or more original executable code elements and the one or more display elements; instructions for receiving from a user, a voice command that includes a request to perform an operation to create new content using the content already displayed in the one or more display elements of the web page, the operation not being in the original executable code elements; instructions for generating a computer program having the one or more content elements as parameters and adding semantic mappings between the computer program to be executed as a command responsive to the received request; adding, using the processor, the command as a new executable code element of the web page; and instructions for executing the command to fulfil the received request by generating new content that is displayed on the web page, wherein generating the new content comprises adding one or more new structured display elements to the web page, the one or more new structured display elements comprising the new content that is dynamically generated based on the content that was already displayed on the web page. ### Claim 14 (depends on 13) The system of claim 13, wherein the received request comprises voice input. ### Claim 15 (depends on 13) The system of claim 13, wherein the one or more new structured content elements comprise new columns added to a table that is already displayed on the web page. ### Claim 16 (depends on 13) The system of claim 13, wherein the one or more mappings comprise domain-specific semantic mappings. ### Claim 17 (depends on 13) The system of claim 13, wherein the injected event monitoring code is associated with specific Document Object Model (DOM) elements within the web-page. ### Claim 18 (depends on 17) The system of claim 17, wherein the computer-readable instructions for generating the command further comprise: instructions for determining if at least one of the one or more parameters is invalid based on the text elements included in the received request; instructions for retrieving one or more events extracted from the one or more event streams; instructions for mapping the text elements to at least one of the one or more received events associated with the DOM elements based on semantic linkages between the text elements and the DOM elements: and instructions for inferring a value of the at least one invalid parameter based on the mapping. ### Claim 19 A computer-program product for multi-modal interaction with structured web page data, the computer-program product comprising a computer-readable storage memory device having program instructions embodied therewith, the program instructions executable by a processor to cause the processor to perform a method comprising: automatically injecting event monitoring code into a web page, the injected event monitoring code comprises identifiers for one or more display elements that include content to be displayed on the web page, and for one or more original executable code elements that include computer programs to operate on the content; receiving one or more event streams generated by the injected event monitoring code, an event stream comprising one or more semantic mappings between the identifiers for the one or more original executable code elements and the one or more display elements; receiving from a user, a voice command that includes a request to perform an operation to create new content using the content displayed in the one or more content elements of the web page, the operation not being in the original executable code elements; generating a computer program having the one or more content elements as parameters and adding semantic mappings between the computer program to be executed as a command responsive to the received request; adding, using the processor, the command as a new executable code element of the web page; and executing the command to fulfil the received request by generating the new content that is displayed on the web page, wherein generating the new content comprises adding one or more new structured display elements to the web page, the one or more new structured content elements comprising the new content that is dynamically generated based on the content that was already displayed on the web page. ### Claim 20 (depends on 19) The computer-program product of claim 19, wherein the one or more mappings comprise domain-specific semantic mappings.