Machine learning (ML) based emotion, identity and voice conversion in audio using virtual domain mixing and fake pair-masking

A method, comprising:

• in an electronic device:
• receiving a source audio associated with a first user;
• receiving a reference-speaker audio associated with a second user;
• receiving a reference-emotion audio associated with a third user;
• applying a set of machine learning (ML) models on the received source audio, the received reference-speaker audio, and the received reference-emotion audio;
• generating a converted audio based on the application of the set of ML models, wherein
• the generated converted audio is associated with content of the source audio, an identity of the second user and is further associated with an emotion corresponding to the third user;
• applying each of a source speaker classifier and a source emotion classifier on the generated converted audio;
• and
• re-training an adversarial model based on the application of each of the source speaker classifier and the source emotion classifier, wherein
• an input audio associated with the first user is converted to an output audio associated with the identity of the second user and associated with the emotion of the third user based on the re-training.

A non-transitory computer-readable medium having stored thereon, computer-executable instructions that when executed by an electronic device, causes the electronic device to execute operations, the operations comprising:

• receiving a source audio associated with a first user;
• receiving a reference-speaker audio associated with a second user;
• receiving a reference-emotion audio associated with a third user;
• applying a set of machine learning (ML) models on the received source audio, the received reference-speaker audio, and the received reference-emotion audio;
• generating a converted audio based on the application of the set of ML models, wherein
• the generated converted audio is associated with content of the source audio, an identity of the second user and is further associated with an emotion corresponding to the third user;
• applying each of a source speaker classifier and a source emotion classifier on the generated converted audio;
• and
• re-training an adversarial model based on the application of each of the source speaker classifier and the source emotion classifier, wherein
• an input audio associated with the first user is converted to an output audio associated with the identity of the second user and associated with the emotion of the third user based on the re-training.