# US 11,907,842 B2 · Deep symbolic validation of information extraction systems > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,907,842 B2 - Document ID: US11907842B2 - Status: Granted - Kind: B2 - Application number: 18/154,149 - Filed: 2023-01-13 - Published: 2023-06-08 - Granted: 2024-02-20 - Earliest claimed: 2019-01-07 - Assignees: Nternational Business Machines Corporation - Applicants: International Business Machines Corporation - Inventors: Alfio Massimiliano Gliozzo; Sarthak Dash; Michael Robert Glass; Mustafa Canim - Web page: https://patentgrep.com/patent/US11907842B2 ## CPC - G06N 3/08: Learning methods - G06N 5/022: Knowledge engineering; Knowledge acquisition - G06F 21/554: {involving event detection and direct action} - G06N 3/042: Knowledge-based neural networks; Logical representations of neural networks - G06N 3/043: based on fuzzy logic, fuzzy membership or fuzzy inference, e.g. adaptive neuro-fuzzy inference systems [ANFIS] - G06N 3/0499: Feedforward networks - G06N 3/0895: Weakly supervised learning, e.g. semi-supervised or self-supervised learning - G06F 2221/034: Test or assess a computer or a system - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract A system comprises a memory that stores computer-executable components; and a processor, operably coupled to the memory, that executes the computer-executable components. The system includes a receiving component that receives a corpus of data; a relation extraction component that generates noisy knowledge graphs from the corpus; and a training component that acquires global representations of entities and relation by training from output of the relation extraction component. ## Claims ### Claim 1 A system, comprising: a memory that stores computer-executable components; a processor, operably coupled to the memory, that executes the computer-executable components stored in the memory, wherein the computer-executable components comprise: a training component that acquires global representations of entities and relation by training from output of a relation extraction component, wherein the relation extraction component generates a set of quads from a corpus of data, wherein the quads have form q= where ei ∈V are entities found in the corpus of data, r ∈R is a finite set of relations and s ∈[0, 1]. ### Claim 2 (depends on 1) The system of claim 1, further comprising a perception component that implements function RelEx(e1; e2, θ(e1; e2)) for one or more relations of the set of relations r E R. ### Claim 3 (depends on 2) The system of claim 2, wherein the perception component also returns a set of quads assessing their confidence from analysis of textual evidence. ### Claim 4 (depends on 3) The system of claim 3, wherein the textual evidence is: RelEx(e1, e2, θ(e1, e2))= ri ∈R, where si is a confidence score for relation ri. ### Claim 5 (depends on 3) The system of claim 3, further comprising a validation component that returns a confidence score for any possible triple such that e1, e2 ∈V and r ∈R. ### Claim 6 (depends on 1) The system of claim 1, wherein a mathematical loss function is implemented to account for confidence associated with triples. ### Claim 7 (depends on 1) The system of claim 1, wherein the training is dependent upon noisy output of relation extraction. ### Claim 8 (depends on 1) The system of claim 1, wherein relation triples identify threats in cybersecurity. ### Claim 9 (depends on 3) The system of claim 3, wherein validation is implemented by using a deep net where a loss function is modified to account for fuzzy truth values provided by output of the perception component. ### Claim 10 A computer-implemented method, comprising: acquiring, by the processor, global representations of entities and relation by training from output of relation extraction, wherein the relation extraction component generates a set of quads from the corpus of data, wherein the quads have form q= where ei ∈V are entities found in the corpus of data, r ∈R is a finite set of relations and s ∈[0, 1]. ### Claim 11 (depends on 10) The computer-implemented method of claim 10, further comprising performing a perception act that implements function RelEx(e1; e2, θ(e1; e2)) for one or more relations of the set of relations r ∈R. ### Claim 12 (depends on 11) The computer-implemented method of claim 11, further comprising returning a set of quads assessing their confidence from analysis of textual evidence. ### Claim 13 (depends on 12) The computer-implemented method of claim 12, wherein the textual evidence is: RelEx(e1, e2, θ(e1, e2))=ri ∈R, where si is a confidence score for relation ri. ### Claim 14 (depends on 13) The method of claim 13, further comprising performing a validation act that returns a confidence score for any possible triple such that e1, e2 ∈V and r ∈R. ### Claim 15 (depends on 10) The method of claim 10, wherein relation triples identify threats in cybersecurity. ### Claim 16 A non-transitory computer program product, comprising a computer readable storage medium having program instructions embodied therewith, the program instructions executable by a processor to cause the processor to: acquire, by a processor, global representations of entities and relation by training from output of the relation extraction, wherein the relation extraction component generates a set of quads from the corpus of data, wherein the quads have form q= where ei ∈V are entities found in the corpus of data, r ∈R is a finite set of relations and s ∈[0, 1]. ### Claim 17 (depends on 16) The non-transitory computer program product of claim 16, wherein the program instructions are executable by the processor to also implement function RelEx(e1; e2, θ(e1; e2)) for one or more relations of the set of relations r ∈R. ### Claim 18 (depends on 17) The non-transitory computer program product of claim 17, wherein program instructions are also executable by the processor to return a set of quads assessing their confidence from analysis of textual evidence. ### Claim 19 (depends on 18) The non-transitory computer program product of claim 18, wherein the textual evidence is: RelEx(e1, e2, θ(e1, e2))=ri ∈R, where si is a confidence score for relation ri. ### Claim 20 (depends on 18) The non-transitory computer program product of claim 18, wherein the program instructions are also executable by the processor to return a confidence score for any possible triple such that e1, e2 ∈V and r ∈R.