Method and apparatus for training model, method and apparatus for synthesizing speech, device and storage medium

An electronic device, comprising:

• one or more processors;
• and
• one or more memories, communicatively connected with the one or more processors,
• the one or more memories collectively storing instructions executable by the one or more processors, and the instructions, when collectively executed by the one or more processors, causing the one or more processors to perform operations, the operations comprising:
• determining a phoneme feature and a prosodic word boundary feature of sample text data;
• inserting a pause character into the phoneme feature according to the prosodic word boundary feature to obtain a combined feature of the sample text data;
• and
• training an initial speech synthesis model according to the combined feature of the sample text data, to obtain a target speech synthesis model, wherein the training the initial speech synthesis model according to the combined feature of the sample text data comprises:
• determining a pause hidden feature distribution according to the combined feature and an acoustic feature of sample audio data, the sample audio data being associated with the sample text data;
• and
• performing unsupervised training on the initial speech synthesis model according to the combined feature and the pause hidden feature distribution, wherein the performing unsupervised training on the initial speech synthesis model according to the combined feature and the pause hidden feature distribution comprises:
• determining a combined feature vector representation according to the combined feature;
• obtaining an output of a first encoder by using the combined feature representation as an input of the first encoder;
• obtaining spectrum features of phonemes and pause characters in the combined feature, by using an output of the first encoder and a spectrum feature of the sample audio data associated with the sample text data as an input of a first attention layer;
• obtaining the pause hidden feature distribution by using spectrum features of the pause characters in the combined feature as an input of a second encoder;
• performing sampling on the pause hidden feature distribution to obtain a hidden feature;
• obtaining an output of a third encoder by using the combined feature representation as an input of the third encoder;
• obtaining a final output of the third encoder by concatenating the output of the third encoder and the hidden feature;
• and
• obtaining an acoustic feature based on the final output of the third encoder.

A non-transitory computer readable storage medium, storing computer instructions, the computer instructions being used to cause a computer to perform operations, the operations comprising:

• determining a phoneme feature and a prosodic word boundary feature of sample text data;
• inserting a pause character into the phoneme feature according to the prosodic word boundary feature to obtain a combined feature of the sample text data;
• and
• training an initial speech synthesis model according to the combined feature of the sample text data, to obtain a target speech synthesis model, wherein the training the initial speech synthesis model according to the combined feature of the sample text data comprises:
• determining a pause hidden feature distribution according to the combined feature and an acoustic feature of sample audio data, the sample audio data being associated with the sample text data;
• and
• performing unsupervised training on the initial speech synthesis model according to the combined feature and the pause hidden feature distribution, wherein the performing unsupervised training on the initial speech synthesis model according to the combined feature and the pause hidden feature distribution comprises:
• determining a combined feature vector representation according to the combined feature;
• obtaining an output of a first encoder by using the combine feature representation as an input of the first encoder;
• obtaining spectrum features of phonemes and pause characters in the combined feature by using an output of the first encoder and a spectrum feature of the sample audio data associated with the sample text data as an input of a first attention layer;
• obtaining the pause hidden feature distribution by using spectrum features of the pause characters in the combined feature as an input of a second encoder;
• performing sampling on the pause hidden feature distribution to obtain a hidden feature;
• obtaining an output of a third encoder by using the combined feature representation as an input of the third encoder;
• obtaining a final output of the third encoder by concatenating the output of the third encoder and the hidden feature;
• and
• obtaining an acoustic feature based on the final output of the third encoder.