# US 12,112,135 B2 · Question answering information completion using machine reading comprehension-based process > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,112,135 B2 - Document ID: US12112135B2 - Status: Granted - Kind: B2 - Application number: 17/449,303 - Filed: 2021-09-29 - Published: 2023-03-30 - Granted: 2024-10-08 - Earliest claimed: 2021-09-29 - Assignees: International Business Machines Corporation - Applicants: International Business Machines Corporation - Inventors: Zhong Fang Yuan; Tong Liu; Chen Gao; Xiang Yu Yang - Web page: https://patentgrep.com/patent/US12112135B2 ## CPC - G06F 40/30: Semantic analysis - G06F 16/3329: Natural language query formulation - G06F 18/2148: {characterised by the process organisation or structure, e.g. boosting cascade} - G06N 3/045: Combinations of networks - G06N 3/092: Reinforcement learning - G06N 5/04: Inference or reasoning models - G06F 40/284: Lexical analysis, e.g. tokenisation or collocates - G06F 40/56: Natural language generation - G06N 3/096: Transfer learning - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS ## Abstract An approach is provided for optimizing a feedback-type question answering process. A training set is constructed to detect missing information of a question. A natural language generation model is trained using the missing information. The natural language generation model is executed to generate a rhetorical question. A response to the rhetorical question is combined with the question to generate an input to a language processor. A new question is generated. The new question is applied to a document library. A final answer is generated. ## Claims ### Claim 1 A computer system comprising: a central processing unit (CPU); a memory coupled to the CPU; and one or more computer readable storage media coupled to the CPU, the one or more computer readable storage media collectively containing instructions that are executed by the CPU via the memory to cause the processor to implement a question answering system process, comprising: constructing, by the processor, a training set to detect missing information of a question received by the processor, the question semantically related to a source of information in a document library; identifying a plurality of components of the question, the components including information terms of the question; generating a plurality of masked questions from the question, including, for each of the masked questions, masking one component of the plurality of components of the question at a time until each of the plurality of components is masked to generate the masked question; applying a reading comprehension algorithm to the each masked question to generate new training data; generating the new training data from a combination of the source of information, the masked component, the question, and an answer assessment generated from a comparison of the source of information and the component of the question that is masked; receiving, by a reinforcement learning (RL) system, the new training data to predict the missing information of the question; training, by the processor, a natural language generation model using the missing information, including, in response to the generating the answer assessment, selecting words about which another question is generated for clarifying the question; executing, by the processor, the natural language generation model to generate the other question clarifying the question using the selected words; combining, by the processor, a response to the other question and the question to generate an input to a language processor; generating, by the language processor, a new question; applying the new question to a document library; and generating a final answer. ### Claim 2 (depends on 1) The computer system of claim 1, wherein constructing the training set includes the computer executing the reading comprehension dataset to according to the reinforcement learning (RL) system to predict the missing information in the question. ### Claim 3 (depends on 2) The computer system of claim 2, wherein the training set includes a Stanford Question Answering Dataset (SQUAD)-type comprehension dataset, and wherein a new dataset is constructed from the SQUAD-type comprehension dataset. ### Claim 4 (depends on 3) The computer system of claim 3, wherein the new dataset is further generated according to a reinforcement learning model to detect the missing information. ### Claim 5 (depends on 1) The computer system of claim 1, wherein the language processor includes executes combination of a Generative Pre-trained Transformer (GPT)-3 model and a text style transfer (TST) algorithm to generate the new question. ### Claim 6 (depends on 5) The computer system of claim 5, wherein the combination of the GPT-3 model and the TST algorithm are performed on contents of the document library. ### Claim 7 (depends on 1) The computer system of claim 1, wherein the document library is part of a knowledge base. ### Claim 8 (depends on 1) The computer system of claim 1, wherein constructing the training set includes: applying, by the processor, a data mask to the each component of the question to cover or redact the information terms of the question; applying, by the processor, the reading comprehension algorithm to the question with the covered or information terms; and searching, by the processor, for information in the document library semantically related to the original question to generate the new training data. ### Claim 9 (depends on 8) The computer system of claim 8, wherein the new training data is generated by a predetermined set of rules establishing a reward score of the answer assessment established by conditions for masking the terms, so that that the new training data includes a combination of the masked word, the question, the reward, the information, and the answer, and wherein the new training data is applied to a reinforcement learning technique of the RL system. ### Claim 10 A feedback-type question answering method, comprising: receiving, by a data processing system, a question from a user, the data processing system comprising at least one processor and at least one memory, the at least one memory comprising instructions which are executed by the at least one processor; constructing, by the at least one processor, a training set to detect missing information of the question received by the at least one processor, the question semantically related to a source of information in a document library; identifying a plurality of components of the question, the components including information terms of the question; generating a plurality of masked questions from the question, including, for each of the masked questions, masking one component of the plurality of components of the question at a time until each of the plurality of components is masked to generate the masked question; applying a reading comprehension algorithm to the each masked question to generate new training data; generating the new training data from a combination of the source of information, the masked component, the question, and an answer assessment generated from a comparison of the source of information and the component of the question that is masked; receiving, by a reinforcement learning (RL) system, the new training data to predict the missing information of the question; training, by the at least one processor, a natural language generation model using the missing information, including, in response to the generating the answer assessment, selecting words about which another question is generated for clarifying the question; executing, by the at least one processor, the natural language generation model to generate the other question clarifying the question using the selected words; combining, by the at least one processor, a response to the other question and the question to generate an input to a language processor; generating, by the language processor, a new question; applying the new question to a document library; and generating a final answer. ### Claim 11 (depends on 10) The feedback-type question answering method of claim 10, wherein constructing the training set includes executing the reading comprehension dataset to according to the reinforcement learning (RL) system to predict the missing information in the question. ### Claim 12 (depends on 11) The feedback-type question answering method of claim 11, wherein the training set includes a Stanford Question Answering Dataset (SQUAD)-type comprehension dataset, and wherein a new dataset is constructed from the SQUAD-type comprehension dataset. ### Claim 13 (depends on 12) The feedback-type question answering method of claim 12, wherein the new dataset is further generated according to a reinforcement learning model to detect the missing information. ### Claim 14 (depends on 10) The feedback-type question answering method of claim 10, wherein the language processor includes executes a Generative Pre-trained Transformer (GPT)-3 model and a text style transfer (TST) algorithm to generate the new question. ### Claim 15 (depends on 14) The feedback-type question answering method of claim 14, wherein the GPT-3 model and the TST algorithm are performed on contents of the document library. ### Claim 16 (depends on 10) The feedback-type question answering method of claim 10, wherein the document library is part of a knowledge base. ### Claim 17 (depends on 10) The feedback-type question answering method of claim 10, wherein constructing the data set includes: applying a data mask to the each component of the question to cover or redact the information terms of the question; applying the reading comprehension algorithm to the question with the covered or information terms; and searching for information in the document library semantically related to the original question to generate the new training data. ### Claim 18 (depends on 17) The feedback-type question answering method of claim 17, wherein the new training data is generated by a predetermined set of rules establishing a reward score established by conditions for masking the terms, so that that the new training data includes a combination of the masked word, the question, the reward, the information, and the answer, and wherein the new training data is applied to a reinforcement learning technique of the RL system. ### Claim 19 A computer program product for optimizing a feedback-type question answering process, the computer program product comprising: one or more non-transitory computer readable storage media having computer readable program code collectively stored on the one or more computer readable storage media, the computer readable program code being executed by a central processing unit (CPU) of a computer system to cause the computer system to perform a method, said method comprising: receiving, by the computer system, a question from a user, the data processing system comprising at least one processor and at least one memory; constructing, by the computer system, a training set to detect missing information of the question received by the at least one processor, the question semantically related to a source of information in a document library; identifying a plurality of components of the question, the components including information terms of the question; generating a plurality of masked questions from the question, including, for each of the masked questions, masking one component of the plurality of components of the question at a time until each of the plurality of components is masked to generate the masked question; applying a reading comprehension algorithm to the each masked question to generate new training data; generating the new training data from a combination of the source of information, the masked component, the question, and an answer assessment generated from a comparison of the source of information and the component of the question that is masked; receiving, by a reinforcement learning (RL) system, the new training data to predict the missing information of the question; training, by the computer system, a natural language generation model using the missing information, including selecting words about which another question is generated for clarifying the question, including, in response to the generating the answer assessment, selecting words about which another question is generated for clarifying the question; executing, by the computer system, the natural language generation model to generate, including selecting words about which another question is generated for clarifying the question using the selected words; combining, by the computer system, a response to the other question and the question to generate an input; generating, by the computer system, a new question; applying, by the computer system, the new question to a document library; and generating, by the computer system, a final answer. ### Claim 20 (depends on 19) The computer program product of claim 19, wherein constructing the training set includes: applying a data mask to the each component of the question to cover or redact information terms of the question; applying the reading comprehension algorithm to the question with the covered or information terms; and searching for information in the document library semantically related to the original question to generate new training data.