# US 12,008,000 B2 · Automated fact checking using iterative knowledge base querying > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,008,000 B2 - Document ID: US12008000B2 - Status: Granted - Kind: B2 - Application number: 17/747,463 - Filed: 2022-05-18 - Published: 2023-12-14 - Granted: 2024-06-11 - Earliest claimed: 2022-05-18 - Assignees: International Business Machines Corporation - Applicants: International Business Machines Corporation - Inventors: G P Shrivatsa Bhargav; Saswati Dana; Dinesh Khandelwal; Dinesh Garg - Web page: https://patentgrep.com/patent/US12008000B2 ## CPC - G06F 16/24553: {of query operations} - G06F 16/3329: Natural language query formulation - G06F 16/9024: {Graphs; Linked lists (G06F16/9027 takes precedence)} - G06F 40/205: Parsing - G06F 40/30: Semantic analysis - G06N 5/022: Knowledge engineering; Knowledge acquisition - G06N 5/04: Inference or reasoning models - G06N 5/046: Forward inferencing; Production systems - G06F 16/24522: {Translation of natural language queries to structured queries} - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS ## Abstract An embodiment includes decomposing a natural language assertion into a natural language question and answer pair that includes an initial question and an initial answer. The embodiment translates the initial question into a structured knowledge graph query and then performs an iterative process comprising iterative querying of a knowledge graph and evaluating of corresponding query responses resulting in respective confidence scores. A first iteration of the iterative process comprises querying of the knowledge graph to retrieve a first predicted answer, then determining whether a degree of similarity between the initial answer and the first predicted answer meets a threshold criterion. If not, the first predicted query is altered and used for querying the knowledge graph in a subsequent iteration of the iterative process. The embodiment also generates an assertion correctness score indicative of a degree of confidence that the assertion is factual using the respective confidence scores. ## Claims ### Claim 1 A computer implemented method comprising: decomposing a natural language assertion into a natural language question and answer pair, the natural language question and answer pair comprising an initial question and an initial answer; translating the initial question into a structured knowledge graph query; performing an iterative process comprising iterative querying of a knowledge graph and evaluating of corresponding query responses resulting in respective confidence scores, wherein a first iteration of the iterative process comprises: querying of the knowledge graph by executing, as a first predicted query, the structured knowledge graph query to retrieve a first predicted answer as a query response from the knowledge graph; determining whether a first confidence score of the respective confidence scores meets a threshold criterion, wherein the first confidence score is indicative of a degree of similarity between the initial answer and the first predicted answer; and altering, responsive to the first confidence score failing to meet the threshold criterion, the first predicted query based on a difference between the initial answer and the first predicted answer, wherein the altering results in an altered predicted query; wherein subsequent iterations of the iterative process comprise querying of the knowledge graph using respective altered predicted queries; and generating an assertion correctness score using the respective confidence scores, wherein the assertion correctness score is indicative of a degree of confidence that the assertion is factual. ### Claim 2 (depends on 1) The method of claim 1, wherein the decomposing of the natural language assertion comprises parsing the natural language assertion into an Abstract Meaning Representation (AMR) graph. ### Claim 3 (depends on 2) The method of claim 2, wherein the decomposing of the natural language assertion further comprises deleting a node of the AMR graph and replacing the thus deleted node with an amr-unknown node. ### Claim 4 (depends on 3) The method of claim 3, wherein the decomposing of the natural language assertion further comprises converting the AMR graph with the amr-unknown node into the initial question. ### Claim 5 (depends on 4) The method of claim 4, wherein the decomposing of the natural language assertion further comprises converting the deleted node into the initial answer. ### Claim 6 (depends on 1) The method of claim 1, wherein the iterative process comprises designating the assertion as true if the predicted answer is equal to the initial answer. ### Claim 7 (depends on 1) The method of claim 1, wherein the iterative process comprises determining that the query response is a subset of the initial answer. ### Claim 8 (depends on 7) The method of claim 7, wherein the iterative process comprises removing, responsive to determining that the query response is a subset of the initial answer, a constraint from the first predicted query for a next iteration. ### Claim 9 (depends on 1) The method of claim 1, wherein the iterative process comprises determining that the query response is a superset of the initial answer. ### Claim 10 (depends on 9) The method of claim 9, wherein the iterative process comprises adding, responsive to determining that the query response is a subset of the initial answer, a constraint to the first predicted query for a next iteration. ### Claim 11 (depends on 1) The method of claim 1, wherein the iterative process comprises determining that a union between the query response and the initial answer is an empty set. ### Claim 12 (depends on 11) The method of claim 11, wherein the iterative process comprises adding a constraint to and removing a constraint from the first predicted query in response to determining that a union between the query response and the initial answer is an empty set. ### Claim 13 A computer program product, the computer program product comprising one or more computer readable storage media, and program instructions collectively stored on the one or more computer readable storage media, the program instructions executable by one or more processors to cause the one or more processors to perform operations comprising: decomposing a natural language assertion into a natural language question and answer pair, the natural language question and answer pair comprising an initial question and an initial answer; translating the initial question into a structured knowledge graph query; performing an iterative process comprising iterative querying of a knowledge graph and evaluating of corresponding query responses resulting in respective confidence scores, wherein a first iteration of the iterative process comprises: querying of the knowledge graph by executing, as a first predicted query, the structured knowledge graph query to retrieve a first predicted answer as a query response from the knowledge graph; determining whether a first confidence score of the respective confidence scores meets a threshold criterion, wherein the first confidence score is indicative of a degree of similarity between the initial answer and the first predicted answer; and altering, responsive to the first confidence score failing to meet the threshold criterion, the first predicted query based on a difference between the initial answer and the first predicted answer, wherein the altering results in an altered predicted query; wherein subsequent iterations of the iterative process comprise querying of the knowledge graph using respective altered predicted queries; and generating an assertion correctness score using the respective confidence scores, wherein the assertion correctness score is indicative of a degree of confidence that the assertion is factual. ### Claim 14 (depends on 13) The computer program product of claim 13, wherein the stored program instructions are stored in a computer readable storage device in a data processing system, and wherein the stored program instructions are transferred over a network from a remote data processing system. ### Claim 15 (depends on 13) The computer program product of claim 13, wherein the stored program instructions are stored in a computer readable storage device in a server data processing system, and wherein the stored program instructions are downloaded in response to a request over a network to a remote data processing system for use in a computer readable storage device associated with the remote data processing system, further comprising: program instructions to meter use of the program instructions associated with the request; and program instructions to generate an invoice based on the metered use. ### Claim 16 (depends on 13) The computer program product of claim 13, wherein the decomposing of the natural language assertion comprises parsing the natural language assertion into an Abstract Meaning Representation (AMR) graph. ### Claim 17 (depends on 16) The computer program product of claim 16, wherein the decomposing of the natural language assertion further comprises deleting a node of the AMR graph and replacing the thus deleted node with an amr-unknown node. ### Claim 18 A computer system comprising one or more processors and one or more computer readable storage media, and program instructions collectively stored on the one or more computer readable storage media, the program instructions executable by the one or more processors to cause the one or more processors to perform operations comprising: decomposing a natural language assertion into a natural language question and answer pair, the natural language question and answer pair comprising an initial question and an initial answer; translating the initial question into a structured knowledge graph query; performing an iterative process comprising iterative querying of a knowledge graph and evaluating of corresponding query responses resulting in respective confidence scores, wherein a first iteration of the iterative process comprises: querying of the knowledge graph by executing, as a first predicted query, the structured knowledge graph query to retrieve a first predicted answer as a query response from the knowledge graph; determining whether a first confidence score of the respective confidence scores meets a threshold criterion, wherein the first confidence score is indicative of a degree of similarity between the initial answer and the first predicted answer; and altering, responsive to the first confidence score failing to meet the threshold criterion, the first predicted query based on a difference between the initial answer and the first predicted answer, wherein the altering results in an altered predicted query; wherein subsequent iterations of the iterative process comprise querying of the knowledge graph using respective altered predicted queries; and generating an assertion correctness score using the respective confidence scores, wherein the assertion correctness score is indicative of a degree of confidence that the assertion is factual. ### Claim 19 (depends on 18) The computer system of claim 18, wherein the decomposing of the natural language assertion comprises parsing the natural language assertion into an Abstract Meaning Representation (AMR) graph. ### Claim 20 (depends on 19) The computer system of claim 19, wherein the decomposing of the natural language assertion further comprises deleting a node of the AMR graph and replacing the thus deleted node with an amr-unknown node.