Personalized cover-voice music synthesis system

A system for cover music synthesis, comprising:

• one or more processors;
• and
• one or more non-transitory computer-readable media storing instructions that, when executed by the one or more processors, cause the system to:
• access a first digital song specified in a playlist that includes one or more digital songs;
• extract, from the first digital song, first musical parameters that characterize music content of the first digital song;
• obtain a first vocal embedding indicative of a first singer, the first vocal embedding representing speech spoken by or vocals sung by the first singer;
• and
• generate, using the first vocal embedding, first synthetic vocals for the first digital song with a vocal synthesis model, the first synthetic vocals being conditioned on the first musical parameters,
• wherein, to obtain the first vocal embedding indicative of the first singer, the instructions further cause the system to:
• access a repository storing a plurality of vocal embeddings, each vocal embedding corresponding to a respective singer;
• apply a clustering algorithm to the plurality of vocal embeddings based on one or both of (i) the first musical parameters and (ii) one or more prosodic parameters used to condition the vocal synthesis model, thereby generating a plurality of clusters of the vocal embeddings;
• select a first cluster from the plurality of clusters;
• and
• select, from the first cluster, the vocal embedding corresponding to the first singer as the first vocal embedding.

A non-transitory computer-readable medium storing instructions that, when executed by one or more processors of a system, cause the system to:

• access a first digital song specified in a playlist that includes one or more digital songs;
• extract, from the first digital song, first musical parameters that characterize music content of the first digital song;
• obtain a first vocal embedding indicative of a first singer, the first vocal embedding representing speech spoken by or vocals sung by the first singer;
• and
• generate, using the first vocal embedding, first synthetic vocals for the first digital song with a vocal synthesis model, the first synthetic vocals being conditioned on the first musical parameters,
• wherein, to obtain the first vocal embedding indicative of the first singer, the instructions further cause the system to:
• access a repository storing a plurality of vocal embeddings, each vocal embedding corresponding to a respective singer;
• apply a clustering algorithm to the plurality of vocal embeddings based on one or both of (i) the first musical parameters and (ii) one or more prosodic parameters used to condition the vocal synthesis model, thereby generating a plurality of clusters of the vocal embeddings;
• select a first cluster from the plurality of clusters;
• and
• select, from the first cluster, the vocal embedding corresponding to the first singer as the first vocal embedding.