# US 10,372,763 B2 · Generating probabilistic annotations for entities and relations using reasoning and corpus-level evidence > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,372,763 B2 - Document ID: US10372763B2 - Status: Granted - Kind: B2 - Application number: 14/797,634 - Filed: 2015-07-13 - Published: 2017-01-19 - Granted: 2019-08-06 - Earliest claimed: 2015-07-13 - Assignees: International Business Machines Corporation - Applicants: International Business Machines Corporation - Inventors: Sachindra Joshi; Raghuram Krishnapuram - Web page: https://patentgrep.com/patent/US10372763B2 ## CPC - G06F 16/93: Document management systems - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract Methods, systems, and computer program products for generating probabilistic annotations for entities and relations using reasoning and corpus-level evidence are provided herein. A method includes determining candidate entities and candidate relationships between entities from text contained within textual documents associated with a given domain; generating questions to support or refute legitimacy of a given one of the candidate entities and/or a given one of the candidate relationships; automatically determining an answer to each of the questions based on analysis of one or more additional items of knowledge pertaining to the given domain; computing a confidence score for each answer; and assigning a confidence value in support of the legitimacy of the given one of the candidate entities and/or the given one of the candidate relationships, wherein said confidence value is based on an aggregation of the confidence scores for the answers to the one or more questions. ## Claims ### Claim 1 A method, comprising: determining (i) one or more candidate entities and (ii) one or more candidate relationships between two or more entities from text contained within a set of textual documents associated with a given domain; generating one or more questions to support or refute legitimacy of a given one of the candidate entities or a given one of the candidate relationships, wherein said one or more questions are generated based on (i) the given one of the candidate entities or the given one of the candidate relationships and (ii) one or more items of knowledge pertaining to the given domain, and wherein said generating is carried out by a query composer component executing on a hardware processor and communicatively linked to a domain knowledge storage component; automatically determining an answer to each of the one or more questions based on analysis of one or more additional items of knowledge pertaining to the given domain; computing a confidence score for each answer, wherein said confidence score is based on (i) a comparison of a given answer to a database of annotated documents associated with the given domain, and (ii) textual information including (a) text surrounding the given one of the candidate entities or the given one of the candidate relationships in the documents, (b) frequency of occurrence of the given one of the candidate entities or the given one of the candidate relationships in the documents, and (c) locations of the given one of the candidate entities or the given one of the candidate relationships in the documents, and wherein said computing is carried out by a confidence assignment component executing on the hardware processor and communicatively linked to an annotated document database; and assigning a confidence value in support of the legitimacy of the given one of the candidate entities or the given one of the candidate relationships, wherein said confidence value is based on an aggregation of (i) the confidence scores for the answers to the one or more questions and (ii) a distinct weight attributed to each of the one or more questions, wherein the distinct weight attributed to each of the one or more questions characterizes a level of importance attributed to the question for confirming a given entity type, and wherein said assigning is carried out by an evidence aggregation component executing on the hardware processor and communicatively linked to the confidence assignment component. ### Claim 2 (depends on 1) The method of claim 1, wherein comparison of the given answer to the database of annotated documents associated with the given domain comprises a determination as to whether a given answer type associated with the given answer matches an expected answer type associated with a corresponding given question. ### Claim 3 (depends on 1) The method of claim 1, wherein said generating the one or more questions comprises implementing a template-based generation method. ### Claim 4 (depends on 1) The method of claim 1, wherein said assigning the confidence value in support of the legitimacy of the given one of the candidate entities or the given one of the candidate relationships comprises implementing a rule-based method. ### Claim 5 (depends on 1) The method of claim 1, wherein said assigning the confidence value in support of the legitimacy of the given one of the candidate entities or the given one of the candidate relationships comprises implementing a machine learning-based method. ### Claim 6 (depends on 1) The method of claim 1, comprising: annotating (i) the one or more determined candidate entities and (ii) the one or more determined candidate relationships in the set of textual documents associated with the given domain. ### Claim 7 (depends on 1) The method of claim 1, comprising: generating a ranking of (i) the one or more determined candidate entities and (ii) the one or more determined candidate relationships based on an estimated legitimacy value. ### Claim 8 (depends on 7) The method of claim 7, wherein the estimated legitimacy value is based on an aggregated score calculated by combining multiple confidence scores associated with answers to multiple questions generated for a given candidate entity. ### Claim 9 (depends on 1) The method of claim 1, comprising: storing the (i) one or more determined candidate entities and (ii) the one or more determined candidate relationships. ### Claim 10 (depends on 9) The method of claim 9, wherein said storing comprises storing (i) the one or more determined candidate entities and (ii) the one or more determined candidate relationships in an inverted index. ### Claim 11 (depends on 9) The method of claim 9, wherein said storing comprises storing (i) the one or more determined candidate entities and (ii) the one or more determined candidate relationships in an extensible markup language index. ### Claim 12 A computer program product, the computer program product comprising a computer readable storage medium having program instructions embodied therewith, the program instructions executable by a computing device to cause the computing device to: determine (i) one or more candidate entities and (ii) one or more candidate relationships between two or more entities from text contained within a set of textual documents associated with a given domain; generate one or more questions to support or refute legitimacy of a given one of the candidate entities or a given one of the candidate relationships, wherein said one or more questions are generated based on (i) the given one of the candidate entities or the given one of the candidate relationships and (ii) one or more items of knowledge pertaining to the given domain, and wherein said generating is carried out by a query composer component executing on a hardware processor and communicatively linked to a domain knowledge storage component; automatically determine an answer to each of the one or more questions based on analysis of one or more additional items of knowledge pertaining to the given domain; compute a confidence score for each answer, wherein said confidence score is based on (i) a comparison of a given answer to a database of annotated documents associated with the given domain, and (ii) textual information including (a) text surrounding the given one of the candidate entities or the given one of the candidate relationships in the documents, (b) frequency of occurrence of the given one of the candidate entities or the given one of the candidate relationships in the documents, and (c) locations of the given one of the candidate entities or the given one of the candidate relationships in the documents, and wherein said computing is carried out by a confidence assignment component executing on the hardware processor and communicatively linked to an annotated document database; and assign a confidence value in support of the legitimacy of the given one of the candidate entities or the given one of the candidate relationships, wherein said confidence value is based on an aggregation of (i) the confidence scores for the answers to the one or more questions and (ii) a distinct weight attributed to each of the one or more questions, wherein the distinct weight attributed to each of the one or more questions characterizes a level of importance attributed to the question for confirming a given entity type, and wherein said assigning is carried out by an evidence aggregation component executing on the hardware processor and communicatively linked to the confidence assignment component. ### Claim 13 (depends on 12) The computer program product of claim 12, wherein said assigning the confidence value in support of the legitimacy of the given one of the candidate entities or the given one of the candidate relationships comprises implementing a rule-based method or a machine learning-based method. ### Claim 14 (depends on 12) The computer program product of claim 12, wherein the program instructions executable by the computing device further cause the computing device to: annotate (i) the one or more determined candidate entities and (ii) the one or more determined candidate relationships in the set of textual documents associated with the given domain. ### Claim 15 A system comprising: a memory; and at least one processor coupled to the memory and configured for: determining (i) one or more candidate entities and (ii) one or more candidate relationships between two or more entities from text contained within a set of textual documents associated with a given domain; generating one or more questions to support or refute legitimacy of a given one of the candidate entities or a given one of the candidate relationships, wherein said one or more questions are generated based on (i) the given one of the candidate entities or the given one of the candidate relationships and (ii) one or more items of knowledge pertaining to the given domain, and wherein said generating is carried out by a query composer component executing on the at least one processor and communicatively linked to a domain knowledge storage component; automatically determining an answer to each of the one or more questions based on analysis of one or more additional items of knowledge pertaining to the given domain; computing a confidence score for each answer, wherein said confidence score is based on (i) a comparison of a given answer to a database of annotated documents associated with the given domain, and (ii) textual information including (a) text surrounding the given one of the candidate entities or the given one of the candidate relationships in the documents, (b) frequency of occurrence of the given one of the candidate entities or the given one of the candidate relationships in the documents, and (c) locations of the given one of the candidate entities or the given one of the candidate relationships in the documents, and wherein said computing is carried out by a confidence assignment component executing on the hardware processor and communicatively linked to an annotated document database; and assigning a confidence value in support of the legitimacy of the given one of the candidate entities or the given one of the candidate relationships, wherein said confidence value is based on an aggregation of (i) the confidence scores for the answers to the one or more questions and (ii) a distinct weight attributed to each of the one or more questions, wherein the distinct weight attributed to each of the one or more questions characterizes a level of importance attributed to the question for confirming a given entity type, and wherein said assigning is carried out by an evidence aggregation component executing on the at least one processor and communicatively linked to the confidence assignment component. ### Claim 16 A method, comprising: determining (i) one or more candidate entities and (ii) one or more candidate relationships between two or more entities from text contained within a set of textual documents associated with a given domain; generating one or more questions to support or refute legitimacy of each of the candidate entities and each of the candidate relationships, wherein each of said one or more questions is generated based on (i) the given candidate entity or the given candidate relationship and (ii) one or more items of knowledge pertaining to the given domain, and wherein said generating is carried out by a query composer component executing on a hardware processor and communicatively linked to a domain knowledge storage component; automatically determining an answer to each of the one or more questions based on analysis of one or more additional items of knowledge pertaining to the given domain; computing a confidence score for each answer, wherein said confidence score is based on (i) a comparison of a given answer to a database of annotated documents associated with the given domain, and (ii) textual information including (a) text surrounding the given one of the candidate entities or the given one of the candidate relationships in the documents, (b) frequency of occurrence of the given one of the candidate entities or the given one of the candidate relationships in the documents, and (c) locations of the given one of the candidate entities or the given one of the candidate relationships in the documents, and wherein said computing is carried out by a confidence assignment component executing on the hardware processor and communicatively linked to an annotated document database; assigning a confidence value in support of the legitimacy of each of the candidate entities and each of the candidate relationships, wherein each confidence value is based on an aggregation of (i) the confidence scores for the answers to the one or more questions and (ii) a distinct weight attributed to each of the one or more questions, wherein the distinct weight attributed to each of the one or more questions characterizes a level of importance attributed to the question for confirming a given entity type, and wherein said assigning is carried out by an evidence aggregation component executing on the hardware processor and communicatively linked to the confidence assignment component; ranking each of the candidate entities and each of the candidate relationships based on the confidence value in support of the legitimacy thereof; and storing each of the candidate entities and each of the candidate relationships based on said ranking. ### Claim 17 (depends on 16) The method of claim 16, wherein comparison of the given answer to the database of annotated documents associated with the given domain comprises a determination as to whether a given answer type associated with the given answer matches an expected answer type associated with the corresponding given question. ### Claim 18 (depends on 16) The method of claim 16, wherein said generating the one or more questions comprises implementing a template-based generation method. ### Claim 19 (depends on 16) The method of claim 16, wherein said assigning a confidence value in support of the legitimacy of each of the candidate entities and each of the candidate relationships comprises implementing a rule-based method or a machine learning-based method. ### Claim 20 (depends on 16) The method of claim 16, comprising: annotating each of the candidate entities and each of the candidate relationships in the set of textual documents associated with the given domain.