Method and apparatus for training encoder
AssigneesHyundai Motor Company, Kia Corporation
InventorSung Woong Hwang
Life of this patent
Estimating the term…
Abstract
A method and an apparatus for training a speech transformation model are provided. The method and the apparatus are capable of generating a natural speech suitable for context and improving accuracy of pronunciation by training the first encoder (e.g., a encoder of the flow-based model) and the second encoder (e.g., a encoder of the Tacotron 2 model) in parallel.
Claims18
A computer implement method for training a speech transformation model, which includes a first encoder, a second encoder, a third encoder, one or more attention modules, and a phoneme duration prediction unit, using dataset including text and speech, the method comprising:
- generating, using the first encoder, first output data from training data generated by preprocessing the text, the first output data related to first features of the text;
- generating, using the second encoder, second output data from the training data generated by preprocessing the text, the second output data related to second features of the text;
- generating, using the third encoder, third output data from the speech corresponding to the text, the third output data related to features of the speech;
- generating, using the one or more attention modules, (i) a first similarity from the first output data and the third output data and (ii) a second similarity from the second output data and the third output data, the first similarity and the second similarity representing similarities between the first and second features of the text and the features of the speech;
- generating, using the phoneme duration prediction unit, a predicted phoneme duration of the text based on a weighted sum of the first output data with a weight based on the first similarity and the second output data with a weight based on the second similarity;
- and
- updating one or more parameters of the speech transformation model based on a loss function to which the at least one of the first output data, the second output data, or the third output data are applied.
An apparatus for training a speech transformation model including a first encoder, a second encoder, a third encoder, one or more attention modules, and a phoneme duration prediction unit, the apparatus comprising:
- a memory storing instructions;
- and
- at least one or more processors configured to execute the instructions to perform operations comprising:
- generating, using the first encoder, first output data from training data generated by preprocessing a text, the first output data related to first features of the text;
- generating, using the second encoder, second output data from the training data generated by preprocessing the text of the training data, the second output data related to second features of the text;
- generating, using the third encoder, third output data from speech corresponding to the text, the third output data related to features of the speech;
- generating, using the one or attention modules, (i) a first similarity from the first output data and the third output data and (ii) a second similarity from the second output data and the third output data, the first similarity and the second similarity representing similarities between the first and second features of the text and the features of the speech;
- generating, using the phoneme duration prediction unit, a predicted phoneme duration of the text based on a weighted sum of the first output data with a weight based on the first similarity and the second output data with a weight based on the second similarity;
- and
- updating one or more parameters of the speech transformation model based on a loss function to which the at least one of the first output data, the second output data, or the third output data are applied.
Dependencies are inferred from the claim text.