# US 12,118,979 B2 · Text-to-speech synthesis system and method > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,118,979 B2 - Document ID: US12118979B2 - Status: Granted - Kind: B2 - Application number: 18/346,694 - Filed: 2023-07-03 - Published: 2023-11-16 - Granted: 2024-10-15 - Earliest claimed: 2018-03-28 - Assignees: Telepathy Labs, Inc. - Applicants: Telepathy Labs, Inc. - Inventors: Piero Perucci; Martin Reber; Vijeta Avijeet - Web page: https://patentgrep.com/patent/US12118979B2 ## CPC - G10L 13/02: Methods for producing synthetic speech; Speech synthesisers - G10L 13/047: Architecture of speech synthesisers - G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU] - G06N 3/08: Learning methods - G06N 3/09: Supervised learning - G10L 13/08: Text analysis or generation of parameters for speech synthesis out of text, e.g. grapheme to phoneme translation, prosody generation or stress or intonation determination - G10L 25/51: for comparison or discrimination - G10L 25/30: using neural networks - G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS ## Abstract A method, computer program product, and computer system for text-to-speech synthesis is disclosed. Synthetic speech data for an input text may be generated. The synthetic speech data may be compared to recorded reference speech data corresponding to the input text. Based on, at least in part, the comparison of the synthetic speech data to the recorded reference speech data, at least one feature indicative of at least one difference between the synthetic speech data and the recorded reference speech data may be extracted. A speech gap filling model may be generated based on, at least in part, the at least one feature extracted. A speech output may be generated based on, at least in part, the speech gap filling model. ## Claims ### Claim 1 A computing system including one or more processors and one or more memories configured to perform operations comprising: comparing synthetic speech data for an input text to recorded reference speech data corresponding to the input text; extracting at least one feature indicative of at least one difference between the synthetic speech data and the recorded reference speech data based on, at least in part, the comparison of the synthetic speech data to the recorded reference speech data; generating a speech gap filling model based on, at least in part, the at least one feature extracted; generating a speech output based on, at least in part, the speech gap filling model; comparing the speech output generated for a second input text to recorded reference speech data corresponding to the second input text; and extracting an updated at least one feature indicative of at least one difference between the speech output generated for the second input text and the recorded reference speech data corresponding to the second input text based on, at least in part, the comparison of the speech output for the second input text to the recorded reference speech data corresponding to the second input text. ### Claim 2 (depends on 1) The computing system of claim 1, wherein generating the speech output comprises: generating an interim set of parameters; processing the interim set of parameters based on, at least in part, the speech gap filling model to generate a final set of parameters; and generating the speech output based on, at least in part, the final set of parameters. ### Claim 3 (depends on 1) The computing system of claim 1, wherein the synthetic speech data generated is based on, at least in part, at least one of a parametric acoustic model and a linguistic model pre-configured for a speaker. ### Claim 4 (depends on 1) The computing system of claim 1, wherein the synthetic speech data generated is further based on, at least in part, the recorded reference speech data pre-recorded by a speaker. ### Claim 5 (depends on 1) The computing system of claim 1 further comprising aligning the synthetic speech data and the recorded reference speech data preceding the comparison. ### Claim 6 (depends on 5) The computing system of claim 5, wherein aligning the synthetic speech data and the recorded reference speech data comprises implementing one or more of pitch shifting, time normalization, and time alignment between the synthetic speech data and the recorded reference speech data. ### Claim 7 (depends on 1) The computing system of claim 1 further comprising training a neural network based on, at least in part, the at least one feature to generate the speech gap filling model. ### Claim 8 (depends on 1) The computing system of claim 1 further comprising updating the speech gap filling model based on, at least in part, the updated at least one feature. ### Claim 9 A computer-implemented method, comprising: comparing synthetic speech data for an input text to recorded reference speech data corresponding to the input text; extracting at least one feature indicative of at least one difference between the synthetic speech data and the recorded reference speech data based on, at least in part, the comparison of the synthetic speech data to the recorded reference speech data; generating a speech gap filling model based on, at least in part, the at least one feature extracted; and generating a speech output based on, at least in part, the speech gap filling model; comparing the speech output generated for a second input text to recorded reference speech data corresponding to the second input text; and extracting an updated at least one feature indicative of at least one difference between the speech output generated for the second input text and the recorded reference speech data corresponding to the second input text based on, at least in part, the comparison of the speech output for the second input text to the recorded reference speech data corresponding to the second input text. ### Claim 10 (depends on 9) The computer-implemented method of claim 9, wherein generating the speech output comprises: generating an interim set of parameters; processing the interim set of parameters based on, at least in part, the speech gap filling model to generate a final set of parameters; and generating the speech output based on, at least in part, the final set of parameters. ### Claim 11 (depends on 9) The computer-implemented method of claim 9, wherein the synthetic speech data generated is based on, at least in part, at least one of a parametric acoustic model and a linguistic model pre-configured for a speaker. ### Claim 12 (depends on 9) The computer-implemented method of claim 9, wherein the synthetic speech data generated is further based on, at least in part, the recorded reference speech data pre-recorded by a speaker. ### Claim 13 (depends on 9) The computer-implemented method of claim 9 further comprising aligning the synthetic speech data and the recorded reference speech data preceding the comparison. ### Claim 14 (depends on 13) The computer-implemented method of claim 13, wherein aligning the synthetic speech data and the recorded reference speech data comprises implementing one or more of pitch shifting, time normalization, and time alignment between the synthetic speech data and the recorded reference speech data. ### Claim 15 (depends on 9) The computer-implemented method of claim 9 further comprising training a neural network based on, at least in part, the at least one feature to generate the speech gap filling model. ### Claim 16 (depends on 9) The computer-implemented method of claim 9 further comprising updating the speech gap filling model based on, at least in part, the updated at least one feature. ### Claim 17 A computer program product residing on a computer readable storage medium having a plurality of instructions stored thereon which, when executed across one or more processors, causes at least a portion of the one or more processors to perform operations comprising: comparing synthetic speech data for an input text to recorded reference speech data corresponding to the input text; extracting at least one feature indicative of at least one difference between the synthetic speech data and the recorded reference speech data based on, at least in part, the comparison of the synthetic speech data to the recorded reference speech data; generating a speech gap filling model based on, at least in part, the at least one feature extracted; generating a speech output based on, at least in part, the speech gap filling model; comparing the speech output generated for a second input text to recorded reference speech data corresponding to the second input text; and extracting an updated at least one feature indicative of at least one difference between the speech output generated for the second input text and the recorded reference speech data corresponding to the second input text based on, at least in part, the comparison of the speech output for the second input text to the recorded reference speech data corresponding to the second input text. ### Claim 18 (depends on 17) The computer program product of claim 17, wherein generating the speech output comprises: generating an interim set of parameters; processing the interim set of parameters based on, at least in part, the speech gap filling model to generate a final set of parameters; and generating the speech output based on, at least in part, the final set of parameters. ### Claim 19 (depends on 17) The computer program product of claim 17, wherein the synthetic speech data generated is based on, at least in part, at least one of a parametric acoustic model and a linguistic model pre-configured for a speaker. ### Claim 20 (depends on 17) The computer program product of claim 17, wherein the synthetic speech data generated is further based on, at least in part, the recorded reference speech data pre-recorded by a speaker. ### Claim 21 (depends on 17) The computer program product of claim 17 further comprising aligning the synthetic speech data and the recorded reference speech data preceding the comparison. ### Claim 22 (depends on 21) The computer program product of claim 21, wherein aligning the synthetic speech data and the recorded reference speech data comprises implementing one or more of pitch shifting, time normalization, and time alignment between the synthetic speech data and the recorded reference speech data. ### Claim 23 (depends on 17) The computer program product of claim 17 further comprising training a neural network based on, at least in part, the at least one feature to generate the speech gap filling model. ### Claim 24 (depends on 17) The computer program product of claim 17 further comprising updating the speech gap filling model based on, at least in part, the updated at least one feature.