Robust direct speech-to-speech translation

A system comprising:

• data processing hardware;
• and
• memory hardware in communication with the data processing hardware and storing instructions that, when executed on the data processing hardware, cause the data processing hardware to execute a direct speech-to-speech translation (S2ST) model, the S2ST model comprising:
• an encoder configured to:
• receive an input speech representation corresponding to an utterance spoken by a source speaker in a first language and in a first voice;
• and
• encode the input speech representation into a hidden feature representation;
• an attention module configured to generate a context vector that attends to the hidden feature representation encoded by the encoder;
• a decoder configured to:
• receive the context vector generated by the attention module;
• and
• predict a phoneme representation corresponding to a translation of the utterance in a second different language, the phoneme representation comprising a sequence of phonemes;
• and
• a synthesizer configured to:
• receive the context vector and the phoneme representation;
• and
• generate a translated synthesized speech representation corresponding to the translation of the utterance spoken in the different second language and in the first voice,
• wherein the S2ST model is trained end-to-end on pairs of parallel source language and target language utterances each in a same voice, the S2ST model trained to not translate input speech representations corresponding to the source language utterances into corresponding translated synthesized speech representations that are in a different voice than the corresponding source language utterances.