# US 12,603,078 B2 · Generating speech data using artificial intelligence techniques > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,603,078 B2 - Document ID: US12603078B2 - Status: Granted - Kind: B2 - Application number: 18/449,809 - Filed: 2023-08-15 - Published: 2025-02-20 - Granted: 2026-04-14 - Earliest claimed: 2023-08-15 - Assignees: International Business Machines Corporation - Applicants: International Business Machines Corporation - Inventors: Zvi Kons; Ron Hoory; Vyacheslav Shechtman; Avihu Dekel - Web page: https://patentgrep.com/patent/US12603078B2 ## CPC - G10L 13/02: Methods for producing synthetic speech; Speech synthesisers - G10L 13/04: Details of speech synthesis systems, e.g. synthesiser structure or memory management - G10L 13/08: Text analysis or generation of parameters for speech synthesis out of text, e.g. grapheme to phoneme translation, prosody generation or stress or intonation determination - G10L 13/10: Prosody rules derived from text; Stress or intonation - G10L 25/30: using neural networks - G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING ## Abstract Methods, systems, and computer program products for generating speech data using artificial intelligence techniques are provided herein. A computer-implemented method includes implementing one or more artificial intelligence techniques in connection with one or more speech synthesis tasks; generating, in multiple sequential portions, at least one sequence of data, comprising one or more of phonetic data and prosodic data, by processing at least one previously generated sequence of data using the one or more artificial intelligence techniques; and generating speech data corresponding to at least a portion of the sequence of data by processing the at least a portion of the sequence of data using at least one artificial intelligence-based speech synthesis model. ## Claims ### Claim 1 A system comprising: a memory configured to store program instructions; and a processor operatively coupled to the memory to execute the program instructions to: implement artificial intelligence techniques in connection with one or more speech synthesis tasks, wherein implementing artificial intelligence techniques comprises combining at least one language model (LM), at least one text-to-speech frontend (TTS-FE) model and at least one text-to-speech prosody (TTS-P) model; generate, in multiple sequential portions, at least one sequence of data comprising phonetic data and prosodic data, by processing at least one previously generated sequence of data using the artificial intelligence techniques, wherein generating the at least one sequence of data comprises: generating one or more phonemes and one or more items of prosodic data associated with one or more output words related to the at least one previously generated sequence of data; using at least a portion of the one or more phonemes and the one or more items of prosodic data to generate one or more hierarchical prosody control (HPC) features comprising at least one of (i) one or more global assessments of at least one of phone rate, pitch, and volume, and (ii) one or more local assessments of at least one of phone rate, pitch, and volume; and generating, as at least part of the at least one sequence of data, one or more vectors based at least in part on the one or more HPC features; and generate speech data corresponding to at least a portion of the at least one sequence of data by processing the at least a portion of the at least one sequence of data using at least one artificial intelligence-based speech synthesis model. ### Claim 2 (depends on 1) The system of claim 1, wherein generating speech data comprises generating one or more speech waveforms corresponding to at least a portion of the at least one sequence of data. ### Claim 3 (depends on 1) The system of claim 1, wherein implementing artificial intelligence techniques comprises converting, using a language model-text-to-speech algorithm adaptor, at least a portion of output from the at least one LM to phonetic data and prosodic data to be used as input for one or more of the at least one TTS-FE model and the at least one TTS-P model. ### Claim 4 (depends on 3) The system of claim 3, wherein implementing artificial intelligence techniques comprises using a language model-text-to-speech algorithm adaptor in conjunction with generating one or more HPC features derived from one or more statistical measurements taken over one or more hierarchical temporal intervals, and normalized to represent one or more speaker-agnostic features, wherein at least a portion of the one or more HPC features comprises at least one of (i) one or more global assessments of at least one of phone rate, pitch, and volume, and (ii) one or more local assessments of at least one of phone rate, pitch, and volume. ### Claim 5 (depends on 1) The system of claim 1, wherein implementing artificial intelligence techniques comprises: modifying at least a portion of the at least one LM using one or more items of the phonetic data; and extracting the one or more items of the phonetic data from at least one set of training text data using the at least one TTS-FE model. ### Claim 6 (depends on 1) The system of claim 1, wherein implementing artificial intelligence techniques comprises: modifying at least a portion of the at least one LM using the one or more items of prosodic data; and extracting the one or more items of prosodic data from at least one set of training text data using the at least one TTS-P model. ### Claim 7 (depends on 1) The system of claim 1, wherein implementing artificial intelligence techniques comprises modifying at least a portion of the at least one LM using one or more items of speech data in connection with at least one automated speech recognition technique and one or more prosodic feature extraction techniques. ### Claim 8 (depends on 1) The system of claim 1, wherein the processor is further operatively coupled to the memory to execute the program instructions to: automatically train at least a portion of the artificial intelligence techniques using at least a portion of the generated speech data. ### Claim 9 A computer program product comprising a non-transitory computer readable storage medium having program instructions embodied therewith, the program instructions executable by a computing device to cause the computing device to: implement artificial intelligence techniques in connection with one or more speech synthesis tasks, wherein implementing artificial intelligence techniques comprises combining at least one language model (LM), at least one text-to-speech frontend (TTS-FE) model and at least one text-to-speech prosody (TTS-P) model; generate, in multiple sequential portions, at least one sequence of data comprising phonetic data and prosodic data, by processing at least one previously generated sequence of data using the artificial intelligence techniques, wherein generating the at least one sequence of data comprises: generating one or more phonemes and one or more items of prosodic data associated with one or more output words related to the at least one previously generated sequence of data; using at least a portion of the one or more phonemes and the one or more items of prosodic data to generate one or more hierarchical prosody control (HPC) features comprising at least one of (i) one or more global assessments of at least one of phone rate, pitch, and volume, and (ii) one or more local assessments of at least one of phone rate, pitch, and volume; and generating, as at least part of the at least one sequence of data, one or more vectors based at least in part on the one or more HPC features; and generate speech data corresponding to at least a portion of the at least one sequence of data by processing the at least a portion of the at least one sequence of data using at least one artificial intelligence-based speech synthesis model. ### Claim 10 (depends on 9) The computer program product of claim 9, wherein implementing artificial intelligence techniques comprises converting, using a language model-text-to-speech algorithm adaptor, at least a portion of output from the at least one LM to phonetic data and prosodic data to be used as input for one or more of the at least one TTS-FE model and the at least one TTS-P model. ### Claim 11 (depends on 10) The computer program product of claim 10, wherein implementing artificial intelligence techniques comprises using a language model-text-to-speech algorithm adaptor in conjunction with generating one or more HPC features derived from one or more statistical measurements taken over one or more hierarchical temporal intervals, and normalized to represent one or more speaker-agnostic features, wherein at least a portion of the one or more HPC features comprises at least one of (i) one or more global assessments of at least one of phone rate, pitch, and volume, and (ii) one or more local assessments of at least one of phone rate, pitch, and volume. ### Claim 12 A computer-implemented method comprising: implementing artificial intelligence techniques in connection with one or more speech synthesis tasks, wherein implementing artificial intelligence techniques comprises combining at least one language model (LM), at least one text-to-speech frontend (TTS-FE) model and at least one text-to-speech prosody (TTS-P) model; generating, in multiple sequential portions, at least one sequence of data comprising phonetic data and prosodic data, by processing at least one previously generated sequence of data using the artificial intelligence techniques, wherein generating the at least one sequence of data comprises: generating one or more phonemes and one or more items of prosodic data associated with one or more output words related to the at least one previously generated sequence of data; using at least a portion of the one or more phonemes and the one or more items of prosodic data to generate one or more hierarchical prosody control (HPC) features comprising at least one of (i) one or more global assessments of at least one of phone rate, pitch, and volume, and (ii) one or more local assessments of at least one of phone rate, pitch, and volume; and generating, as at least part of the at least one sequence of data, one or more vectors based at least in part on the one or more HPC features; and generating speech data corresponding to at least a portion of the at least one sequence of data by processing the at least a portion of the at least one sequence of data using at least one artificial intelligence-based speech synthesis model; wherein the method is carried out by at least one computing device. ### Claim 13 (depends on 12) The computer-implemented method of claim 12, wherein implementing artificial intelligence techniques comprises converting, using a language model-text-to-speech algorithm adaptor, at least a portion of output from the at least one LM to phonetic data and prosodic data to be used as input for one or more of the at least one TTS-FE model and the at least one TTS-P model. ### Claim 14 (depends on 13) The computer-implemented method of claim 13, wherein implementing artificial intelligence techniques comprises using a language model-text-to-speech algorithm adaptor in conjunction with generating one or more HPC features derived from one or more statistical measurements taken over one or more hierarchical temporal intervals, and normalized to represent one or more speaker-agnostic features, wherein at least a portion of the one or more HPC features comprises at least one of (i) one or more global assessments of at least one of phone rate, pitch, and volume, and (ii) one or more local assessments of at least one of phone rate, pitch, and volume. ### Claim 15 (depends on 12) The computer-implemented method of claim 12, wherein implementing artificial intelligence techniques comprises: modifying at least a portion of the at least one LM using the one or more items of prosodic data; and extracting the one or more items of prosodic data from at least one set of training text data using the at least one TTS-P model. ### Claim 16 (depends on 12) The computer-implemented method of claim 12, wherein implementing artificial intelligence techniques comprises modifying at least a portion of the at least one LM using one or more items of speech data in connection with at least one automated speech recognition technique and one or more prosodic feature extraction techniques. ### Claim 17 (depends on 12) The computer-implemented method of claim 12, further comprising: automatically training at least a portion of the artificial intelligence techniques using at least a portion of the generated speech data. ### Claim 18 (depends on 9) The computer program product of claim 9, wherein implementing artificial intelligence techniques comprises: modifying at least a portion of the at least one LM using one or more items of the phonetic data; and extracting the one or more items of the phonetic data from at least one set of training text data using the at least one TTS-FE model. ### Claim 19 (depends on 9) The computer program product of claim 9, wherein implementing artificial intelligence techniques comprises: modifying at least a portion of the at least one LM using the one or more items of prosodic data; and extracting the one or more items of prosodic data from at least one set of training text data using the at least one TTS-P model. ### Claim 20 (depends on 9) The computer program product of claim 9, wherein implementing artificial intelligence techniques comprises modifying at least a portion of the at least one LM using one or more items of speech data in connection with at least one automated speech recognition technique and one or more prosodic feature extraction techniques.