Streaming speech-to-speech model with automatic speaker turn detection

A system comprising:

• data processing hardware;
• and
• memory hardware storing instructions that when executed on data processing hardware causes the data processing hardware to perform operations comprising:
• receiving, as input to a speech-to-speech (S2S) model, a sequence of acoustic frames corresponding to an utterance spoken by a user in streaming audio captured by a client device associated with the user, wherein the S2S model is configured to convert input acoustic frames directly into output acoustic frames corresponding to synthesized speech without performing speech recognition on the input acoustic frames;
• and
• at each of a plurality of output steps:
• generating, by an audio encoder of the S2S model, a higher order feature representation for a corresponding acoustic frame in the sequence of acoustic frames;
• determining, by a turn detector of the S2S model, based on the higher order feature representation generated by the audio encoder at the corresponding output step, whether the utterance is at a breakpoint at the corresponding output step;
• and
• when the turn detector determines that the utterance is at the breakpoint, synthesizing a sequence of output audio frames output by a speech decoder of the S2S model into a time-domain audio waveform of synthesized speech representing the utterance spoken by the user, each output audio frame in the sequence of output audio frames based on a corresponding one of the higher order feature representations generated by the audio encoder until the corresponding output step when the turn detector determines the utterance is at the breakpoint,
• wherein the turn detector is disposed between the audio encoder and a speech decoder of the S2S model, and wherein determining whether the utterance is at the breakpoint comprises:
• receiving, by the turn detector from the audio encoder, the higher order feature representation at the corresponding output step;
• and
• generating, by the turn detector, a turn output indicative of whether the utterance is at the breakpoint based on the higher order feature representation at the corresponding output step.