Techniques for improved zero-shot voice conversion with a conditional disentangled sequential variational auto-encoder

An apparatus for voice conversion using a conditional disentangled sequential variational auto-encoder (C-DSVAE), the apparatus comprising:

• at least one memory configured to store program code;
• and
• at least one processor configured to read the program code and operate as instructed by the program code, the program code including:
• receiving code configured to cause the at least one processor to receive input speech segments;
• first encoding code configured to cause the at least one processor to encode the input speech segments via a shared encoder to generate a speaker embedding and a content embedding;
• second encoding code configured to cause the at least one processor to encode a posterior distribution of the speaker embedding via a speaker encoder and encode a posterior distribution of the content embedding via a content encoder to obtain encoded results;
• enabling code configured to cause the at least one processor to enable a content bias, and reshape the content embedding by preserving phonetic information using the content bias sampled from the posterior distribution;
• and
• generating code configured to cause the at least one processor to generate a reconstructed speech output based on the encoded results and the reshaped content embedding,
• wherein a total loss of the voice conversion is based on a reconstruction loss between the input speech segments and the reconstructed speech output, a prior distribution and the posterior distribution of the speaker embedding, and a Kullback-Leibler (KL)-Divergence between a conditional prior and the posterior distribution of the content embedding, and
• wherein the content bias is forced alignment, and the content bias is concatenated to prior modeling during a training process, and
• wherein during the training process, the content embedding and a target embedding are concatenated when inferencing to obtain a voice conversion speech output, and the concatenated results of the content embedding and the target embedding are provided to a decoder to generate the reconstructed speech in a form of a spectrogram.

A non-transitory computer-readable medium storing instructions, the instructions comprising:

• one or more instructions that, when executed by at least one processor of an apparatus for voice conversion using a conditional disentangled sequential variational auto-encoder (C-DSVAE) storing instructions that, cause the at least one processor to:
• receive input speech segments;
• encode the input speech segments via a shared encoder to generate a speaker embedding and a content embedding;
• encode a posterior distribution of the speaker embedding via a speaker encoder and encode a posterior distribution of the content embedding via a content encoder to obtain encoded results;
• enable a content bias, and reshape the content embedding by preserving phonetic information using the content bias sampled from the posterior distribution;
• and
• generate a reconstructed speech output based on the encoded results and the reshaped content embedding to generate the reconstructed speech output,
• wherein a total loss of the voice conversion is based on a reconstruction loss between the input speech segments and the reconstructed speech output, a prior distribution and the posterior distribution of the speaker embedding, and a Kullback-Leibler (KL)-Divergence between a conditional prior and the posterior distribution of the content embedding,
• wherein the content bias is forced alignment, and the content bias is concatenated to prior modeling during a training process, and
• wherein during the training process, the content embedding and a target embedding are concatenated when inferencing to obtain a voice conversion speech output, and the concatenated results of the content embedding and the target embedding are provided to a decoder to generate the reconstructed speech in a form of a spectrogram.