Method and system for building text-to-speech voice from diverse recordings

A system comprising:

• one or more processors;
• memory;
• and
• machine-readable instructions stored in the memory, that upon execution by the one or more processors cause the system to carry out operations including:
• extracting speech features from a plurality of recorded reference speech utterances of a reference speaker to generate a reference set of reference-speaker vectors,
• for each respective plurality of recorded colloquial speech utterances of a respective colloquial speaker of multiple colloquial speakers, extracting speech features from the recorded colloquial speech utterances of the respective colloquial speaker to generate a respective set of colloquial-speaker vectors,
• for each respective set of colloquial-speaker vectors, replacing each colloquial-speaker vector of the respective set of colloquial-speaker vectors with a respective, optimally-matched reference-speaker vector from among the reference set of reference-speaker vectors, wherein the respective, optimally-matched reference-speaker vector is identified by matching under a transform that compensates for differences in speech between the reference speaker and the respective colloquial speaker,
• aggregating the replaced colloquial-speaker vectors of all the respective sets of colloquial-speaker vectors into an aggregate set of conditioned speaker vectors,
• providing the aggregate set of conditioned speaker vectors to a text-to-speech (TTS) system, and
• training the TTS system using the provided aggregate set of conditioned speaker vectors.

An article of manufacture including a non-transitory computer-readable storage medium having stored thereon program instructions that, upon execution by one or more processors of a system, cause the system to perform operations comprising:

• extracting speech features from a plurality of recorded reference speech utterances of a reference speaker to generate a reference set of reference-speaker vectors;
• for each respective plurality of recorded colloquial speech utterances of a respective colloquial speaker of multiple colloquial speakers, extracting speech features from the recorded colloquial speech utterances of the respective colloquial speaker to generate a respective set of colloquial-speaker vectors;
• for each respective set of colloquial-speaker vectors, replacing each colloquial-speaker vector of the respective set of colloquial-speaker vectors with a respective, optimally-matched reference-speaker vector from among the reference set of reference-speaker vectors, wherein the respective, optimally-matched reference-speaker vector is identified by matching under a transform that compensates for differences in speech between the reference speaker and the respective colloquial speaker;
• aggregating the replaced colloquial-speaker vectors of all the respective sets of colloquial-speaker vectors into an aggregate set of conditioned speaker vectors;
• providing the aggregate set of conditioned speaker vectors to a text-to-speech (TTS) system implemented on one or more computing devices;
• and
• training the TTS system using the provided aggregate set of conditioned speaker vectors.