# US 2021/0280164 A1 · Learnable speed control for speech synthesis > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2021/0280164 A1 - Document ID: US20210280164A1 - Status: Published application - Kind: A1 - Application number: 16/807,801 - Filed: 2020-03-03 - Published: 2021-09-09 - Earliest claimed: 2020-03-03 - Assignees: Tencent America LLC - Applicants: Tencent America LLC - Inventors: Chengzhu YU; Dong YU - Web page: https://patentgrep.com/patent/US20210280164A1 ## CPC - G10L 13/033: Voice editing, e.g. manipulating the voice of the synthesiser - G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU] - G06N 3/0455: Auto-encoder networks; Encoder-decoder networks - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/09: Supervised learning - G10L 13/047: Architecture of speech synthesisers - G10L 13/06: Elementary speech units used in speech synthesisers; Concatenation rules - G10L 25/18: the extracted parameters being spectral information of each sub-band - G10L 25/24: the extracted parameters being the cepstrum - G06N 3/044: Recurrent networks, e.g. Hopfield networks - G06N 3/045: Combinations of networks - G06N 3/088: Non-supervised learning, e.g. competitive learning - G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS ## Abstract A method, computer program, and computer system is provided for synthesizing speech at one or more speeds. A context associated with one or more phonemes corresponding to a speaking voice is encoded, and the one or more phonemes are aligned to one or more target acoustic frames based on the encoded context. One or more mel-spectrogram features are recursively generated from the aligned phonemes and target acoustic frames, and a voice sample corresponding to the speaking voice is synthesized using the generated mel-spectrogram features. ## Claims ### Claim 1 A method of synthesizing speech at one or more speeds, comprising: encoding, by a computer, a context associated with one or more phonemes corresponding to a speaking voice; aligning, by the computer, the one or more phonemes to one or more target acoustic frames based on the encoded context; recursively generating, by the computer, one or more mel-spectrogram features from the aligned phonemes and the target acoustic frames; and synthesizing, by the computer, a voice sample at a given speed corresponding to the speaking voice using the generated mel-spectrogram features. ### Claim 2 (depends on 1) The method of claim 1, wherein the encoding comprises: receiving a sequence of the one or more phonemes; and outputting a sequence of one or more hidden states containing a sequential representation associated with the received sequence of phonemes. ### Claim 3 (depends on 2) The method of claim 2, wherein the aligning the one or more phonemes to one or more target acoustic frames comprises: concatenating the output sequence of hidden states with information corresponding to the speaking voice; applying dimension reduction to the concatenated output sequence using a fully connected layer; expanding the dimension-reduced output sequence based on a rate associated with each phoneme; and aligning the expanded output sequence to the target acoustic frames. ### Claim 4 (depends on 3) The method of claim 3, further comprising concatenating one or more frame-aligned hidden states with a frame level, a root mean square error value, and a relative position associated with every frame. ### Claim 5 (depends on 4) The method of claim 4, wherein the rate of each phoneme is obtained from a force alignment performed on one or more input phonemes and one or more acoustic features. ### Claim 6 (depends on 1) The method of claim 1, wherein the generating the one or more mel-spectrogram features based on the aligned frames comprises: computing an attention context from one or more encoded hidden states aligned with the one or more target acoustic frames; and applying a CBHG technique to the computed attention context. ### Claim 7 (depends on 6) The method of claim 6, wherein a loss value associated with the mel-spectrogram is minimized. ### Claim 8 (depends on 1) The method of claim 1, wherein the generating the one or more mel-spectrogram features is performed by a recursive neural network. ### Claim 9 (depends on 8) The method of claim 8, wherein the inputs to the recursive neural network comprise a sequence of the one or more phonemes, a rate associated with each of the one or more phonemes, a root mean square error value, and an identity associated with a speaker. ### Claim 10 (depends on 1) The method of claim 1, wherein the voice sample is synthesized without parallel data and without changing content associated with the speaking voice. ### Claim 11 A computer system for synthesizing speech at one or more speeds, the computer system comprising: one or more computer-readable non-transitory storage media configured to store computer program code; and one or more computer processors configured to access said computer program code and operate as instructed by said computer program code, said computer program code including: encoding code configured to cause the one or more computer processors to encode a context associated with one or more phonemes corresponding to a speaking voice; aligning code configured to cause the one or more computer processors to align the one or more phonemes to one or more target acoustic frames based on the encoded context; generating code configured to cause the one or more computer processors to recursively generate one or more mel-spectrogram features from the aligned phonemes and the target acoustic frames; and synthesizing code configured to cause the one or more computer processors to synthesize a voice sample at a given speed corresponding to the speaking voice using the generated mel-spectrogram features. ### Claim 12 (depends on 11) The computer system of claim 11, wherein the encoding code comprises: receiving code configured to cause the one or more computer processors to receive a sequence of the one or more phonemes; and outputting code configured to cause the one or more computer processors to output a sequence of one or more hidden states containing a sequential representation associated with the received sequence of phonemes. ### Claim 13 (depends on 12) The computer system of claim 12, wherein the aligning code comprises: concatenating code configured to cause the one or more computer processors to concatenate the output sequence of hidden states with information corresponding to the speaking voice; applying code configured to cause the one or more computer processors to apply dimension reduction to the concatenated output sequence using a fully connected layer; expanding code configured to cause the one or more computer processors to expand the dimension-reduced output sequence based on a rate associated with each phoneme; and aligning code configured to cause the one or more computer processors to align the expanded output sequence to the target acoustic frames. ### Claim 14 (depends on 13) The computer system of claim 13, wherein the concatenating code configured to cause the one or more computer processors to concatenate one or more frame-aligned hidden states with a frame level, a root mean square error value, and a relative position associated with every frame. ### Claim 15 (depends on 14) The computer system of claim 14, wherein the rate of each phoneme is obtained from a force alignment performed on one or more input phonemes and one or more acoustic features. ### Claim 16 (depends on 11) The computer system of claim 11, wherein the generating code comprises: computing code configured to cause the one or more computer processors to compute an attention context from one or more encoded hidden states aligned with the one or more target acoustic frames; and applying code configured to cause the one or more computer processors to apply a CBHG technique to the computed attention context. ### Claim 17 (depends on 11) The computer system of claim 11, wherein the generating the one or more mel-spectrogram features is performed by a recursive neural network. ### Claim 18 (depends on 17) The computer system of claim 17, wherein the inputs to the recursive neural network comprise a sequence of the one or more phonemes, a rate associated with each of the one or more phonemes, a root mean square error value, and an identity associated with a speaker. ### Claim 19 (depends on 11) The computer system of claim 11, wherein the voice sample is synthesized without parallel data and without changing content associated with the speaking voice. ### Claim 20 A non-transitory computer readable medium having stored thereon a computer program for synthesizing speech at one or more speeds, the computer program configured to cause one or more computer processors to: encode a context associated with one or more phonemes corresponding to a speaking voice; align the one or more phonemes to one or more target acoustic frames based on the encoded context; recursively generate one or more mel-spectrogram features from the aligned phonemes and the target acoustic frames; and synthesize a voice sample at a given speed corresponding to the speaking voice using the generated mel-spectrogram features.