End-to-end speech diarization via iterative speaker embedding

A system comprising

• data processing hardware;
• memory hardware in communication with the data processing hardware and storing instructions, that when executed by the data processing hardware, cause the data processing hardware to perform operations comprising:
• receiving an input audio signal corresponding to utterances spoken by multiple speakers;
• encoding the input audio signal into a sequence of T temporal embeddings, each temporal embedding associated with a corresponding time step and representing speech content extracted from the input audio signal at the corresponding time step;
• after receiving the input audio signal, during each iteration of a plurality of iterations each corresponding to a respective speaker of the multiple speakers, selecting a respective speaker embedding for the respective speaker of a corresponding iteration by:
• receiving a previously selected speaker embedding comprising an average of each respective speaker embedding previously selected during previous iterations;
• for each respective temporal embedding in the sequence of T temporal embeddings, determining, based on the previously selected speaker embedding, a probability that the respective temporal embedding includes a presence of voice activity by a single new speaker for which a speaker embedding was not previously selected during any previous iteration;
• determining that the probability satisfies a confidence threshold;
• and
• based on determining that the probability satisfies the confidence threshold, selecting a respective one of the temporal embeddings from among the sequence of T temporal embeddings as the respective speaker embedding for the respective speaker of the corresponding iteration, the selected respective one of the temporal embeddings associated with the highest probability for the presence of voice activity by the single new speaker not selected during any previous iteration;
• and
• at each time step, predicting a respective voice activity indicator for each respective speaker of the multiple speakers based on the respective speaker embeddings selected during the plurality of iterations and the respective temporal embedding associated with the corresponding time step, the respective voice activity indicator indicating whether a voice of the respective speaker is active or inactive at the corresponding time step.