# US 2014/0195884 A1 · System and method for automatically detecting and interactively displaying information about entities, activities, and events from multiple-modality natural language sources > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2014/0195884 A1 - Document ID: US20140195884A1 - Status: Published application - Kind: A1 - Application number: 13/543,157 - Filed: 2012-07-06 - Published: 2014-07-10 - Earliest claimed: 2012-06-11 - Assignees: International Business Machines Corporation - Inventors: VITTORIO CASTELLI; Radu Florian; Xiaoqiang Luo; Hema Raghavan - Web page: https://patentgrep.com/patent/US20140195884A1 ## CPC - G06F 40/40: Processing or translation of natural language (natural language analysis G06F40/20; semantic analysis G06F40/30) - G06F 16/285: {Clustering or classification} - G06F 16/345: {Summarisation for human users} - G06F 40/103: Formatting, i.e. changing of presentation of documents (automatic justification G06F40/189; automatic line break hyphenation G06F40/191) - G06F 40/295: Named entity recognition - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract A method for automatically extracting and organizing information by a processing device from a plurality of data sources is provided. A natural language processing information extraction pipeline that includes an automatic detection of entities is applied to the data sources. Information about detected entities is identified by analyzing products of the natural language processing pipeline. Identified information is grouped into equivalence classes containing equivalent information. At least one displayable representation of the equivalence classes is created. An order in which the at least one displayable representation is displayed is computed. A combined representation of the equivalence classes that respects the order in which the displayable representation is displayed is produced. ## Claims ### Claim 1 A non-transitory computer program storage device embodying instructions executable by a processor to interactively display information about entities, activities and events from multiple-modality natural language sources, the non-transitory computer program storage device comprising storage memory configured to store: an information extraction module having instruction code for downloading document content from text and audio/video, for parsing the document content, for detecting mentions, for co-referencing, for cross-document co-referencing and for extracting relations; an information gathering module having instruction code for extracting acquaintances, biography and involvement in events from the information extraction module; and an information display module having instruction code for displaying information from the information gathering module. ### Claim 2 (depends on 1) The non-transitory computer program storage device of claim 1, wherein the information extraction module further comprises instruction code for transcribing audio from video sources and for translating non-English transcribed audio into English text. ### Claim 3 (depends on 1) The non-transitory computer program storage device of claim 1, wherein the information extraction module further comprises instruction code for clustering mentions under a same entity and for linking entity clusters across documents. ### Claim 4 (depends on 1) The non-transitory computer program storage device of claim 1, wherein the information gathering module further comprises instruction code for inputting a sentence and an entity and extracting specific information about the entity from the sentence. ### Claim 5 (depends on 1) The non-transitory computer program storage device of claim 1, wherein the information display module further comprises instruction code for grouping results into non-redundant sets, sorting the non-redundant sets, producing a brief description of each set, selecting a representative snippet for each set, highlighting the portions of the snippet that contain information pertaining to a specific tab, constructing navigation hyperlinks to other pages, and generating data used to graphically represent tab content. ### Claim 6 A non-transitory computer program storage device embodying instructions executable by a processor to automatically extract and organize information from a plurality of data sources, the non-transitory computer program storage device comprising storage memory configured to store: instruction code for applying to the data sources a natural language processing information extraction pipeline that includes an automatic detection of entities; instruction code for identifying information about detected entities by analyzing products of the natural language processing pipeline; instruction code for grouping identified information into equivalence classes containing equivalent information; instruction code for creating at least one displayable representation of the equivalence classes; instruction code for computing an order in which the at least one displayable representation is displayed; and instruction code for producing a combined representation of the equivalence classes that respects an order in which said displayable representation is displayed. ### Claim 7 (depends on 6) The non-transitory computer program storage device of claim 6, wherein each equivalence class comprises a collection of items, each item comprising of a span of text extracted from a document, together with a specification of information about a desired entity derived from the span of text. ### Claim 8 (depends on 6) The non-transitory computer program storage device of claim 6, wherein computing an order in which said displayable representations are displayed further comprises randomly computing the order. ### Claim 9 (depends on 6) The non-transitory computer program storage device of claim 6, wherein grouping identified information into equivalence classes further comprises assigning each identified information to a separate equivalence class. ### Claim 10 (depends on 6) The non-transitory computer program storage device of claim 6, wherein grouping identified information into equivalence classes further comprises: computing a representative instance of each equivalence class; ensuring that representative instances of different classes are not redundant with respect to each other; and ensuring that instances of each equivalence class are redundant with respect to the representative instance of said equivalence class.