Longform speaker diarization by prompting multimodal LLM with chunk-wise in-context self-enrollment

A computer-implemented method executed on data processing hardware that causes the data processing hardware to perform operations comprising:

• receiving, as input to a joint speech recognition and speaker diarization model, a prompt comprising a textual diarization request and corresponding audio data characterizing a conversation between multiple speakers;
• generating, using a tokenizer of the joint speech recognition and speaker diarization model, a sequence of audio encoding chunks based on the corresponding audio data;
• and
• for each respective audio encoding chunk of the sequence of audio encoding chunks, using a trained large language model (LLM) of the joint speech recognition and speaker diarization model:
• generating corresponding diarization results based on the respective audio encoding chunk and the textual diarization request, the corresponding diarization results comprising a speech recognition result that comprises one or more predicted terms, each respective predicted term associated with a corresponding speaker token representing a predicted identity of a respective speaker that spoke the respective predicted term, the trained LLM conditioned on a prior audio cohort generated by the trained LLM for a prior audio encoding chunk, the prior audio cohort providing reference audio data of one or more of the multiple speakers that spoke prior to the respective audio encoding chunk, wherein the trained LLM uses the prior audio cohort to determine whether the respective audio encoding chunk includes speech from any of the one or more speakers that spoke prior to the respective audio encoding chunk;
• and
• generating, based on the corresponding diarization results, a new audio cohort for the respective audio encoding chunk.