# US 2019/0377791 A1 · Natural language generation pattern enhancement > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2019/0377791 A1 - Document ID: US20190377791A1 - Status: Published application - Kind: A1 - Application number: 16/003,540 - Filed: 2018-06-08 - Published: 2019-12-12 - Earliest claimed: 2018-06-08 - Assignees: International Business Machines Corporation - Applicants: International Business Machines Corporation - Inventors: Alaa Abou Mahmoud; Paul R. Bastide; Matthew E. Broomhall; Robert E. Loredo; Fang Lu - Web page: https://patentgrep.com/patent/US20190377791A1 ## CPC - G06F 40/30: Semantic analysis - G06F 40/35: Discourse or dialogue representation - G06F 16/3344: {using natural language analysis} - G06F 40/211: Syntactic parsing, e.g. based on context-free grammar [CFG] or unification grammars - G06F 40/226: Validation - G06F 40/268: Morphological analysis - G06F 40/56: Natural language generation - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract A computer-implemented method, computer system, and computer program product for improving a natural language generation knowledge base is provided. The method may include detecting user input comprising a natural language expression, generating an erroneous natural language response based on the user input and the knowledge base, determining user feedback corresponding to the erroneous natural language response, wherein the user feedback comprises an indication of an end-user reaction to the erroneous natural language response, determining an improvable performance metric with respect to the knowledge base, and updating the knowledge base based on the improvable performance metric, wherein the knowledge base comprises an explicit model of language corresponding to the erroneous natural language response, and wherein updating the knowledge base comprises updating the explicit model of language based on the user input, the erroneous natural language response, and the user feedback. ## Claims ### Claim 1 A computer-implemented method for improving a natural language generation knowledge base, comprising: detecting user input comprising a natural language expression; generating an erroneous natural language response based on the user input and the knowledge base; determining user feedback corresponding to the erroneous natural language response, wherein the user feedback comprises an indication of an end-user reaction to the erroneous natural language response; determining an improvable performance metric with respect to the knowledge base, wherein the improvable performance metric is based on a scoring of the erroneous natural language response; and updating the knowledge base based on the improvable performance metric, wherein the knowledge base comprises an explicit model of language corresponding to the erroneous natural language response, and wherein updating the knowledge base comprises updating the explicit model of language based on the user input, the erroneous natural language response, and the user feedback. ### Claim 2 (depends on 1) The computer-implemented method of claim 1, wherein the erroneous natural language response violates one or more rules of syntax, semantics, morphology, and orthography of the natural language. ### Claim 3 (depends on 1) The computer-implemented method of claim 1, wherein determining the user feedback comprises prompting the end-user for the user feedback. ### Claim 4 (depends on 1) The computer-implemented method of claim 1, wherein determining the user feedback comprises performing end-user attention monitoring. ### Claim 5 (depends on 4) The computer-implemented method of claim 4, wherein the end-user attention monitoring comprises eye-tracking of the end-user with respect to a display instance of NLG-generated texts of the erroneous natural language response. ### Claim 6 (depends on 1) The computer-implemented method of claim 1, wherein the improvable performance metric is determined based on the user input, the erroneous natural language response, and the user feedback. ### Claim 7 A computer system for improving a natural language generation knowledge base, the computer system comprising: one or more computer processors, one or more computer-readable storage media, and program instructions stored on one or more of the computer-readable storage media for execution by at least one of the one or more computer processors, the program instructions, when executed by the at least one of the one or more computer processors, causing the computer system to perform a method comprising: detecting user input comprising a natural language expression; generating an erroneous natural language response based on the user input and the knowledge base; determining user feedback corresponding to the erroneous natural language response, wherein the user feedback comprises an indication of an end-user reaction to the erroneous natural language response; determining an improvable performance metric with respect to the knowledge base, wherein the improvable performance metric is based on a scoring of the erroneous natural language response; and updating the knowledge base based on the improvable performance metric, wherein the knowledge base comprises an explicit model of language corresponding to the erroneous natural language response, and wherein updating the knowledge base comprises updating the explicit model of language based on the user input, the erroneous natural language response, and the user feedback. ### Claim 8 (depends on 7) The computer system of claim 7, wherein the erroneous natural language response violates one or more rules of syntax, semantics, morphology, and orthography of the natural language. ### Claim 9 (depends on 7) The computer system of claim 7, wherein determining the user feedback comprises prompting the end-user for the user feedback. ### Claim 10 (depends on 7) The computer system of claim 7, wherein determining the user feedback comprises performing end-user attention monitoring. ### Claim 11 (depends on 10) The computer system of claim 10, wherein the end-user attention monitoring comprises eye-tracking of the end-user with respect to a display instance of NLG-generated texts of the erroneous natural language response. ### Claim 12 (depends on 7) The computer system of claim 7, wherein the improvable performance metric is determined based on the user input, the erroneous natural language response, and the user feedback. ### Claim 13 A computer program product for improving a natural language generation knowledge base, the computer program product comprising: one or more computer-readable storage devices and program instructions stored on at least one of the one or more computer-readable storage devices for execution by at least one or more computer processors of a computer system, the program instructions, when executed by the at least one of the one or more computer processors, causing the computer system to perform a method comprising: detecting user input comprising a natural language expression; generating an erroneous natural language response based on the user input and the knowledge base; determining user feedback corresponding to the erroneous natural language response, wherein the user feedback comprises an indication of an end-user reaction to the erroneous natural language response; determining an improvable performance metric with respect to the knowledge base, wherein the improvable performance metric is based on a scoring of the erroneous natural language response; and updating the knowledge base based on the improvable performance metric, wherein the knowledge base comprises an explicit model of language corresponding to the erroneous natural language response, and wherein updating the knowledge base comprises updating the explicit model of language based on the user input, the erroneous natural language response, and the user feedback. ### Claim 14 (depends on 13) The computer program product of claim 13, wherein the erroneous natural language response violates one or more rules of syntax, semantics, morphology, and orthography of the natural language. ### Claim 15 (depends on 13) The computer program product of claim 13, wherein determining the user feedback comprises prompting the end-user for the user feedback. ### Claim 16 (depends on 13) The computer program product of claim 13, wherein determining the user feedback comprises performing end-user attention monitoring. ### Claim 17 (depends on 16) The computer program product of claim 16, wherein the end-user attention monitoring comprises eye-tracking of the end-user with respect to a display instance of NLG-generated texts of the erroneous natural language response. ### Claim 18 (depends on 13) The computer program product of claim 13, wherein the improvable performance metric is determined based on the user input, the erroneous natural language response, and the user feedback.