# US 12,010,260 B2 · Detecting synthetic sounds in call audio > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,010,260 B2 - Document ID: US12010260B2 - Status: Granted - Kind: B2 - Application number: 17/453,431 - Filed: 2021-11-03 - Published: 2023-05-04 - Granted: 2024-06-11 - Earliest claimed: 2021-11-03 - Assignees: Capital One Services, LLC - Applicants: Capital One Services, LLC - Inventors: Meredith L. Critzer; Vahid Khanagha; Joshua Edwards; Mia Rodriguez; Tyler Maiman; Aysu Ezen Can; Alexander Lin; Michael Mossoba; Guadalupe Bonilla; Feng Qiu - Web page: https://patentgrep.com/patent/US12010260B2 ## CPC - H04M 3/2281: {Call monitoring, e.g. for law enforcement purposes; Call tracing; Detection or prevention of malicious calls} - G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue - G10L 21/028: using properties of sound source - G10L 25/51: for comparison or discrimination - G10L 25/69: for evaluating synthetic or decoded voice signals - H04M 2201/40: using speech recognition - H04M 2203/301: Management of recordings - H04M: TELEPHONIC COMMUNICATION (circuits for controlling other apparatus via a telephone cable and not involving telephone switching apparatus G08) - G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING ## Abstract In some implementations, a system may capture audio from a call between a calling device and a called device. The system may filter the captured audio to generate a background audio layer. The system may generate an audio footprint that is a representation of sound in the background audio layer. The system may determine that the audio footprint includes a triggering sound footprint based on one or more audio characteristics of the audio footprint. The system may detect synthetic sound based on the audio footprint and after determining that the audio footprint includes the triggering sound footprint, wherein the synthetic sound is indicative of a sound recording. The system may transmit a notification to one or more devices associated with the call based on detecting the synthetic sound. ## Claims ### Claim 1 A system for detecting synthetic sounds in call audio, the system comprising: one or more memories; and one or more processors, communicatively coupled to the one or more memories, configured to: capture audio from a call between a calling device and a called device; filter, based on one or more audio characteristics of the captured audio, the captured audio to remove one or more voices; generate a background audio layer based on filtering the captured audio to remove the one or more voices; generate an audio footprint that is a representation of sound in the background audio layer; determine that the background audio layer includes a triggering sound pattern based on analyzing the audio footprint; detect a synthetic sound pattern in the background audio layer based on determining that the background audio layer includes the triggering sound pattern, wherein the synthetic sound pattern is detected based on at least one of: a first portion of the audio footprint sufficiently matching a second portion of the audio footprint, or comparing a portion of the audio footprint to a plurality of stored audio footprints based on a plurality of priority indicators corresponding to the plurality of stored audio footprints and the portion of the audio footprint sufficiently matching a stored audio footprint, wherein the stored audio footprint is not generated from the captured audio; and transmit a notification to one or more devices associated with the call based on detecting the synthetic sound pattern. ### Claim 2 (depends on 1) The system of claim 1, wherein the one or more processors, to detect the synthetic sound pattern, are configured to: detect the synthetic sound pattern based on the first portion of the audio footprint sufficiently matching the second portion of the audio footprint, wherein one of the first portion or the second portion includes the triggering sound pattern, and wherein a duration of the triggering sound pattern satisfies a threshold. ### Claim 3 (depends on 1) The system of claim 1, wherein the one or more processors, to detect the synthetic sound pattern, are configured to: detect the synthetic sound pattern based on the portion of the audio footprint sufficiently matching the stored audio footprint, wherein the portion of the audio footprint includes the triggering sound pattern, and wherein a duration of the triggering sound pattern satisfies a threshold. ### Claim 4 (depends on 1) The system of claim 1, wherein the one or more processors are further configured to: determine a category of the triggering sound pattern; and identify the plurality of stored audio footprints, including the stored audio footprint, based on the category of the triggering sound pattern. ### Claim 5 (depends on 1) The system of claim 1, wherein the one or more processors are further configured to: determine an order for comparing the portion of the audio footprint to the plurality of stored audio footprints based on the plurality of priority indicators corresponding to the plurality of stored audio footprints, wherein each priority indicator indicates a quantity of times that a corresponding stored audio footprint was sufficiently matched; wherein the one or more processors, to detect the synthetic sound pattern, are configured to: compare the audio footprint to one or more stored audio footprints, of the plurality of stored audio footprints, based on the order; and detect the synthetic sound pattern based on the portion of the audio footprint sufficiently matching the stored audio footprint, wherein the stored audio footprint is included in the plurality of stored audio footprints; and wherein the one or more processors are further configured to: modify a priority indicator associated with the stored audio footprint based on detecting the synthetic sound pattern. ### Claim 6 (depends on 1) The system of claim 1, wherein the one or more processors, to transmit the notification to the one or more devices associated with the call, are configured to: transmit the notification to a device associated with a called party, that is using the called device to participate in the call, based on a determination that the call is ongoing. ### Claim 7 (depends on 1) The system of claim 1, wherein the one or more processors, to transmit the notification to the one or more devices associated with the call, are configured to: transmit the notification to a device associated with a user for which a record was updated in connection with the call. ### Claim 8 (depends on 1) The system of claim 1, wherein the one or more processors are further configured to: determine whether the call is ongoing or has ended; and identify the one or more devices to which the notification is to be transmitted based on determining whether the call is ongoing or has ended. ### Claim 9 (depends on 1) The system of claim 1, wherein the one or more processors are further configured to: identify a change to a record that was made in connection with the call; and reverse the change to the record based on detecting the synthetic sound pattern. ### Claim 10 A method for detecting synthetic audio in call audio, comprising: capturing, by a system, audio from a call between a calling device and a called device; filtering, by the system and based on one or more audio characteristics of the captured audio, the captured audio to generate a background audio layer; generating, by the system, an audio footprint that is a representation of sound in the background audio layer; determining, by the system, that the audio footprint includes a triggering sound footprint based on one or more audio characteristics of the audio footprint; detecting, by the system, synthetic sound based on the audio footprint and after determining that the audio footprint includes the triggering sound footprint, wherein the synthetic sound is indicative of a sound recording, and wherein the synthetic sound is detected based on at least one of: a first portion of the audio footprint sufficiently matching a second portion of the audio footprint, or comparing a portion of the audio footprint to a plurality of stored audio footprints based on a plurality of priority indicators corresponding to the plurality of stored audio footprints and the portion of the audio footprint sufficiently matching a stored audio footprint, wherein the stored audio footprint is not generated from the captured audio; and transmitting, by the system, a notification to one or more devices associated with the call based on detecting the synthetic sound. ### Claim 11 (depends on 10) The method of claim 10, wherein filtering the captured audio to generate the background audio layer comprises: removing, from the captured audio, all audio originating from the called device; and removing a voice from audio originating from the calling device. ### Claim 12 (depends on 10) The method of claim 10, wherein detecting the synthetic sound includes determining that the audio footprint includes a third portion and a fourth portion that are inconsistent with one another. ### Claim 13 (depends on 10) The method of claim 10, wherein detecting the synthetic sound includes determining that the audio footprint includes a third portion, indicative of a muted sound, and a fourth portion indicative of an unmuted sound other than voice audio. ### Claim 14 (depends on 10) The method of claim 10, wherein detecting the synthetic sound includes determining that the audio footprint includes a characteristic indicative of recorded audio. ### Claim 15 (depends on 10) The method of claim 10, further comprising: determining an order for comparing the portion of the audio footprint to the plurality of stored audio footprints based on the plurality of priority indicators corresponding to the plurality of stored audio footprints, wherein each priority indicator indicates a quantity of times that a corresponding stored audio footprint was sufficiently matched. ### Claim 16 (depends on 10) The method of claim 10, further comprising: modifying a priority indicator associated with the stored audio footprint based on detecting the synthetic sound. ### Claim 17 A non-transitory computer-readable medium storing a set of instructions, the set of instructions comprising: one or more instructions that, when executed by one or more processors of a system, cause the system to: capture audio from a call between a calling device and a called device; filter, based on one or more audio characteristics of the captured audio, the captured audio to generate a background audio layer; generate an audio footprint that is a representation of sound in the background audio layer; determine that the captured audio includes synthetic sound based on the audio footprint, wherein the synthetic sound is indicative of a sound recording, and wherein the synthetic sound is detected based on at least one of: a first portion of the audio footprint sufficiently matching a second portion of the audio footprint, or comparing a portion of the audio footprint to a plurality of stored audio footprints based on a plurality of priority indicators corresponding to the plurality of stored audio footprints and the portion of the audio footprint sufficiently matching a stored audio footprint, wherein the stored audio footprint is not generated from the captured audio; and transmit a notification to one or more devices associated with the call based on determining that the captured audio includes the synthetic sound. ### Claim 18 (depends on 17) The non-transitory computer-readable medium of claim 17, wherein the one or more instructions further cause the system to: determine that the audio footprint includes a triggering sound footprint based on one or more audio characteristics of the audio footprint; and wherein the one or more instructions, that cause the system to determine that the captured audio includes the synthetic sound, cause the system to: determine that the captured audio includes the synthetic sound after determining that the audio footprint includes the triggering sound footprint. ### Claim 19 (depends on 17) The non-transitory computer-readable medium of claim 17, wherein the one or more instructions further cause the system to: determine an order for comparing the portion of the audio footprint to the plurality of stored audio footprints based on the plurality of priority indicators corresponding to the plurality of stored audio footprints, wherein each priority indicator indicates a quantity of times that a corresponding stored audio footprint was sufficiently matched. ### Claim 20 (depends on 17) The non-transitory computer-readable medium of claim 17, wherein the one or more instructions further cause the system to: modify a priority indicator associated with the stored audio footprint based on detecting the synthetic sound.