Instantaneous Learning in Text-To-Speech During Dialog

A system comprising:

• data processing hardware;
• and
• memory hardware in communication with the data processing hardware, the memory hardware storing instructions that when executed on the data processing hardware cause the data processing hardware to perform operations comprising:
• receiving a user pronunciation of a particular word present in a query spoken by a user;
• receiving a text-to-speech (TTS) pronunciation of the same particular word that is present in a TTS input, the TTS input comprising a textual representation of a response to the query, and the TTS pronunciation of the particular word is different than the user pronunciation of the particular word;
• obtaining user pronunciation-related features associated with the user pronunciation of the particular word;
• obtaining TTS pronunciation-related features associated with the TTS pronunciation of the particular word;
• generating, as output from a pronunciation decision model configured to receive the user pronunciation-related features and the TTS pronunciation-related features as input, a pronunciation decision selecting the one of the user pronunciation of the particular word or the TTS pronunciation of the particular word that is associated with a highest confidence for use in TTS audio;
• and
• providing, for audible output from a user device associated with the user, the TTS audio comprising a synthesized speech representation of the response to the query using the one of the user pronunciation for the particular word or the TTS pronunciation for the particular word that was selected by the pronunciation decision output from the pronunciation decision model.