# US 12,217,762 B1 · System and method for detecting synthetic speech based on prosody analysis > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,217,762 B1 - Document ID: US12217762B1 - Status: Granted - Kind: B1 - Application number: 18/735,525 - Filed: 2024-06-06 - Granted: 2025-02-04 - Earliest claimed: 2024-06-06 - Assignees: Corsound AI Ltd - Applicants: Corsound AI Ltd - Inventors: Denys Shyrman - Web page: https://patentgrep.com/patent/US12217762B1 ## CPC - G10L 17/26: Recognition of special voice characteristics, e.g. for use in lie detectors; Recognition of animal voices - G10L 25/30: using neural networks - G06N 3/045: Combinations of networks - G10L 13/047: Architecture of speech synthesisers - G10L 13/10: Prosody rules derived from text; Stress or intonation - G10L 17/02: Preprocessing operations, e.g. segment selection; Pattern representation or modelling, e.g. based on linear discriminant analysis [LDA] or principal components; Feature selection or extraction - G10L 17/04: Training, enrolment or model building - G10L 17/06: Decision making techniques; Pattern matching strategies - G10L 17/18: Artificial neural networks; Connectionist approaches - G10L 25/18: the extracted parameters being spectral information of each sub-band - G10L 25/51: for comparison or discrimination - G10L 25/69: for evaluating synthetic or decoded voice signals - G10L 15/26: Speech to text systems (G10L15/08 takes precedence) - G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS ## Abstract System and method detecting synthetic speech may include, using a processor: training a prosody extractor by: providing a training speech sample to an encoder-decoder (codec) model to generate a channel degraded speech sample; providing the channel degraded speech sample to a prosody extractor to extract a prosody embedding; providing to a speech synthesis model the prosody embedding, a codec embedding representing the codec model, speaker identity information and a text representation to generate a spectrogram of the training speech sample; and training the speech synthesis model and the prosody extractor using a loss function defined on the spectrogram generated by the speech synthesis model compared with a spectrogram of the channel degraded speech sample; and using the trained prosody extractor to detect synthetic speech. ## Claims ### Claim 1 A method for detecting synthetic speech, the method comprising, using a processor: training a prosody extractor comprising a first neural network by: generating a channel degraded speech sample by providing a training speech sample to an encoder-decoder (codec) model, wherein the codec model comprises a second neural network that represents effects of a transmission channel on the speech sample, and wherein the channel degraded speech sample comprises a spectrogram that is degraded as if the training speech sample has been transferred through the transmission channel represented by the codec model; generating a prosody embedding by providing the channel degraded speech sample to a prosody extractor; generating a spectrogram of the training speech sample by providing to a speech synthesis model the prosody embedding, a codec embedding representing the codec model, speaker identity information and a text representation, wherein the speech synthesis model comprises a third neural network; training the speech synthesis model and the prosody extractor using a loss function defined based on the spectrogram generated by the speech synthesis model compared with a spectrogram of the channel degraded speech sample; receiving an examined speech sample; determining whether the examined speech sample includes synthetic speech using the trained prosody extractor; and providing a notice in case the examined speech sample is determined to include synthetic speech, wherein the notice comprises an indication that the examined speech sample is suspected as being a synthetic voice sample. ### Claim 2 (depends on 1) The method of claim 1, wherein using the trained prosody extractor to determine whether the examined speech sample includes synthetic speech comprises: providing a reference speech sample to the trained prosody extractor to generate a prosody embedding of the reference speech sample; providing an examined speech sample to the trained prosody extractor to generate a prosody embedding of examined speech sample; comparing the prosody embedding of the reference speech sample with the prosody embedding of examined speech sample; and determining whether the examined speech sample is authentic or synthetic based on the comparison. ### Claim 3 (depends on 2) The method of claim 2, wherein determining whether the examined speech sample is authentic or synthetic based on the comparison comprises: calculating a similarity measure between prosody embedding of the examined speech sample and the prosody embedding of the reference speech sample; and validating the measure of similarity against a threshold. ### Claim 4 (depends on 1) The method of claim 1, comprising generating the text representation by: providing the speech sample to a pretrained automatic speech recognition (ASR) network to output the text representation. ### Claim 5 (depends on 4) The method of claim 4, wherein the text representation comprises a sequence of vector representations of pieces of phonemes in the training speech sample. ### Claim 6 (depends on 1) The method of claim 1, comprising generating the speaker identity information by providing the speech sample to a pretrained speaker identity extractor to extract speaker identity embedding, wherein the speaker identity information is the speaker identity embedding. ### Claim 7 (depends on 1) The method of claim 1, wherein providing the channel degraded speech sample to the prosody extractor comprises providing the channel degraded speech sample to a recording device model to generate a channel and recording device degraded speech sample, and providing the channel and recording device degraded speech sample to the prosody extractor; and wherein training the prosody extractor comprises providing a recording device embedding representing the recording device to the speech synthesis model with the prosody embedding, the speaker identity embedding and the text representation for generating the spectrogram of the training speech sample. ### Claim 8 (depends on 1) The method of claim 1, wherein the speech synthesis model is Tacotron2 model. ### Claim 9 A method for detecting synthetic speech, the method comprising, using a processor: training a prosody extractor comprising a first neural network by: generating a recording device degraded speech sample by providing a training speech sample to a recording device model, wherein the recording device model comprises a second neural network that represents effects of a recording device on the speech sample, and wherein the recording device degraded speech sample comprises a spectrogram that is degraded as if the training speech sample has been recorded by the recording device represented by the codec model; generating a prosody embedding by providing the recording device degraded speech sample to a prosody extractor; generating a spectrogram of the training speech sample by providing to a speech synthesis model the prosody embedding, a recording device embedding representing the recording device model, speaker identity information and a text representation, wherein the speech synthesis model comprises a third neural network; and training the speech synthesis model and the prosody extractor using a loss function defined on the spectrogram generated by the speech synthesis model compared with a spectrogram of the channel degraded speech sample; receiving an examined speech sample; determining whether the examined speech sample includes synthetic speech using the trained prosody extractor; and providing a notice in case the examined speech sample is determined to include synthetic speech, wherein the notice comprises an indication that the examined speech sample is suspected as being a synthetic voice sample. ### Claim 10 (depends on 9) The method of claim 9, wherein using the trained prosody extractor to determine whether the examined speech sample includes synthetic speech comprises: providing a reference speech sample to the trained prosody extractor to generate a prosody embedding of the reference speech sample; providing an examined speech sample to the trained prosody extractor to generate a prosody embedding of examined speech sample; comparing the prosody embedding of the reference speech sample with the prosody embedding of examined speech sample; and determining whether the examined speech sample is authentic or synthetic based on the comparison. ### Claim 11 (depends on 9) The method of claim 9, comprising generating the text representation by: providing the speech sample to a pretrained automatic speech recognition (ASR) network to output the text representation. ### Claim 12 (depends on 11) The method of claim 11, wherein the text representation comprises a sequence of vector representations of pieces of phonemes in the training speech sample. ### Claim 13 (depends on 9) The method of claim 9, comprising generating the speaker identity information by providing the speech sample to a pretrained speaker identity extractor to extract speaker identity embedding, wherein the speaker identity information is the speaker identity embedding. ### Claim 14 A system for detecting synthetic speech, the system comprising: a memory; and a processor configured to: train a prosody extractor comprising a first neural network by: generating a channel degraded speech sample by providing a training speech sample to an encoder-decoder (codec) model, wherein the codec model comprises a second neural network that represents effects of a transmission channel on the speech sample, and wherein the channel degraded speech sample comprises a spectrogram that is degraded as if the training speech sample has been transferred through the transmission channel represented by the codec model; generating a prosody embedding by providing the channel degraded speech sample to a prosody extractor; generate a spectrogram of the training speech sample by providing to a speech synthesis model the prosody embedding, a codec embedding representing the codec model, speaker identity information and a text representation, wherein the speech synthesis model comprises a third neural network; and train the speech synthesis model and the prosody extractor using a loss function defined on the spectrogram generated by the speech synthesis model compared with a spectrogram of the channel degraded speech sample; receive an examined speech sample; determine whether the examined speech sample includes synthetic speech using the trained prosody extractor; and provide a notice in case the examined speech sample is determined to include synthetic speech, wherein the notice comprises an indication that the examined speech sample is suspected as being a synthetic voice sample. ### Claim 15 (depends on 14) The system of claim 14, wherein the processor is configured to use the trained prosody extractor to determine whether the examined speech sample includes synthetic speech by: providing a reference speech sample to the trained prosody extractor to generate a prosody embedding of the reference speech sample; providing an examined speech sample to the trained prosody extractor to generate a prosody embedding of examined speech sample; comparing the prosody embedding of the reference speech sample with the prosody embedding of examined speech sample; and determining whether the examined speech sample is authentic or synthetic based on the comparison. ### Claim 16 (depends on 15) The system of claim 15, wherein the processor is configured to determine whether the examined speech sample is authentic or synthetic based on the comparison by: calculating a similarity measure between prosody embedding of the examined speech sample and the prosody embedding of the reference speech sample; and validating the measure of similarity against a threshold. ### Claim 17 (depends on 14) The system of claim 14, wherein the processor is configured to generate the text representation by: providing the speech sample to a pretrained automatic speech recognition (ASR) network to output the text representation. ### Claim 18 (depends on 17) The system of claim 17, wherein the text representation comprises a sequence of vector representations of pieces of phonemes in the training speech sample. ### Claim 19 (depends on 14) The system of claim 14, wherein the processor is configured to generate the speaker identity information by providing the speech sample to a pretrained speaker identity extractor to extract speaker identity embedding, wherein the speaker identity information is the speaker identity embedding. ### Claim 20 (depends on 14) The system of claim 14, wherein the processor is configured to provide the channel degraded speech sample to the prosody extractor by providing the channel degraded speech sample to a recording device model to generate a channel and recording device degraded speech sample, and providing the channel and recording device degraded speech sample to the prosody extractor; and wherein the processor is configured to train the prosody extractor by providing a recording device embedding representing the recording device to the speech synthesis model together with the prosody embedding, the speaker identity embedding and the text representation for generating the spectrogram of the training speech sample.