# US 10,120,919 B2 · System and method for multi-modal audio mining of telephone conversations > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,120,919 B2 - Document ID: US10120919B2 - Status: Granted - Kind: B2 - Application number: 15/412,992 - Filed: 2017-01-23 - Published: 2017-05-25 - Granted: 2018-11-06 - Earliest claimed: 2007-02-15 - Assignees: Global Tel*Link Corporation - Applicants: Global Tel*Link Corporation - Inventors: Andreas M. Olligschlaeger; Jim Rokosky - Web page: https://patentgrep.com/patent/US10120919B2 ## CPC - G06F 16/254: {Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses} - G06F 16/283: {Multi-dimensional databases or data warehouses, e.g. MOLAP or ROLAP} - G06F 16/685: {using automatically derived transcript of audio data, e.g. lyrics (speech recognition G10L15/00)} - G06F 16/686: {using information manually generated, e.g. tags, keywords, comments, title or artist information, time, location or usage information, user ratings} - G10L 15/26: Speech to text systems (G10L15/08 takes precedence) - H04M 3/2281: {Call monitoring, e.g. for law enforcement purposes; Call tracing; Detection or prevention of malicious calls} - H04M 3/42221: {Conversation recording systems (at the subscriber's set H04M1/656)} - H04M 2201/40: using speech recognition - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING - H04M: TELEPHONIC COMMUNICATION (circuits for controlling other apparatus via a telephone cable and not involving telephone switching apparatus G08) ## Abstract A system and method for the automated monitoring of inmate telephone calls as well as multi-modal search, retrieval and playback capabilities for said calls. A general term for such capabilities is multi-modal audio mining. The invention is designed to provide an efficient means for organizations such as correctional facilities to identify and monitor the contents of telephone conversations and to provide evidence of possible inappropriate conduct and/or criminal activity of inmates by analyzing monitored telephone conversations for events, including, but not limited to, the addition of third parties, the discussion of particular topics, and the mention of certain entities. ## Claims ### Claim 1 A system for multi-modal audio mining of telephone data, the system comprising: one or more circuits and/or processors configured to: retrieve a data record from a call database; augment the retrieved data record with subscriber information and spatial information; analyze the augmented data record after it has been augmented to verify that each data element of the augmented data record is formatted consistent with a relevant format of a plurality of application specific data warehouses; map each data element within the augmented data record to the plurality of application specific data warehouses; and provide a multi-modal query and visualization of the augmented data record to identify events of interest, wherein the augmented data record includes the spatial information, the subscriber information, a transcript of the monitored telephone conversation, a plurality of characteristics of the monitored telephone conversation and the retrieved data that includes audio of the monitored telephone conversation and metadata information of the monitored telephone conversation. ### Claim 2 (depends on 1) The system of claim 1, wherein the retrieved data record includes superfluous information that includes at least one of leading and trailing blanks and non-alphanumeric characters from strings. ### Claim 3 (depends on 1) The system of claim 1, wherein the spatial information includes geographic information relating to a called telephone number. ### Claim 4 (depends on 1) The system of claim 1, wherein each of the application specific data warehouses is a specialized database that stores information of a common type. ### Claim 5 (depends on 1) The system of claim 1, wherein the one or more circuits and/or processors are configured to map each data element by: storing each data element in a respective one of the plurality of application specific data warehouses; and generating and maintaining a master table for the data record that includes each data element of the augmented data record, and a corresponding specialized data warehouse in which each data element is stored. ### Claim 6 (depends on 1) The system of claim 1, wherein the one or more circuits and/or processors are further configured to: remove superfluous information from the retrieved data record; standardize data elements within the retrieved data record; and transcribe the monitored telephone conversation, detect a plurality of characteristics of the monitored telephone conversation, and associate the detected plurality of characteristics with a transcript of the monitored telephone conversation. ### Claim 7 A system for data-mining a monitored telephone conversation, the system comprising: a data transformer configured to receive data from a data source, the received data including audio of a monitored telephone conversation and metadata information of the monitored telephone conversation; a data augmenter configured to augment the received data with telephone subscriber data and spatial information data; a speech recognizer configured to transcribe the monitored telephone conversation, detect a plurality of characteristics of the monitored telephone conversation, and associate the detected plurality of characteristics with a transcript of the monitored telephone conversation; and a data mapper configured to generate a data record for the monitored telephone conversation and store the data record to a multimedia data warehouse, the data record including at least the transcript of the monitored telephone conversation, the plurality of characteristics of the monitored telephone conversation and the augmented received data that includes the audio, the metadata information, the telephone subscriber data and the spatial information data. ### Claim 8 (depends on 7) The system of claim 7, wherein the spatial information data includes geographic location information of a called number. ### Claim 9 (depends on 7) The system of claim 7, wherein the multimedia data warehouse includes a plurality of application specific databases, and wherein the data mapper is further configured to map each data element within the data record to the plurality of application specific databases. ### Claim 10 (depends on 9) The system of claim 9, wherein the data mapper is further configured to map each data element by: storing each data element in a respective one of the plurality of application specific databases; and generating and maintaining a master table for the data record that includes each data element of the data record, and a corresponding specialized data warehouse in which each data element is stored. ### Claim 11 (depends on 7) The system of claim 7, further comprising: a data normalization processor configured to atomize the data record for the monitored telephone conversation with respect to the multimedia data warehouse; and a link creation processor configured to create one or more logical links between the data record for the monitored telephone conversation and one or more data records within the multimedia data warehouse. ### Claim 12 (depends on 11) The system of claim 11, wherein the data normalization processor is configured to utilize a matching algorithm to atomize the data record for the monitored telephone conversation with respect to the multimedia data warehouse. ### Claim 13 A method for multi-modal audio mining of telephone data, the method comprising: retrieving a data record from a call database; augmenting the retrieved data record with subscriber information and spatial information; analyzing the augmented data record to verify that each data element of the augmented data record is formatted consistent with a relevant format of a plurality of application specific data warehouses; mapping each data element within the augmented data record to the plurality of application specific data warehouses; and providing a multi-modal query and visualization of the augmented data record to identify events of interest, wherein the augmented data record includes the spatial information, the subscriber information, a transcript of a monitored telephone conversation, a plurality of characteristics of the monitored telephone conversation and the retrieved data that includes audio of the monitored telephone conversation and metadata information of the monitored telephone conversation. ### Claim 14 (depends on 13) The method of claim 13, wherein the retrieved data record includes superfluous information including at least one of leading and trailing blanks and non-alphanumeric characters from strings. ### Claim 15 (depends on 13) The method of claim 13, wherein the spatial information includes geographic information relating to a called telephone number. ### Claim 16 (depends on 13) The method of claim 13, wherein each of the application specific data warehouses is a specialized database that stores information of a common type. ### Claim 17 (depends on 13) The method of claim 13, wherein the mapping each data element includes: storing each data element in a respective one of the plurality of application specific data warehouses; and generating and maintaining a master table for the data record that includes each data element of the data record, and a corresponding specialized data warehouse in which each data element is stored.