Context-aware real-time speaker prioritization using acoustic and linguistic embeddings

One or more non-transitory computer-readable media storing instructions executable to perform operations, the operations comprising:

• receiving an audio input signal including speech from one or more speakers;
• encoding audio input signal frames at a speech-to-text encoder to generate linguistic token data and acoustic speaker embeddings;
• identifying the one or more speakers based on the acoustic speaker embeddings;
• determining, at a context fusion engine, a priority score for each of the one or more speakers;
• selecting, by a focus selector, a primary speaker based on the priority scores;
• generating, by an enhancement engine, a spectral emphasis mask based on at least the selected primary speaker and the acoustic speaker embeddings;
• and
• applying the spectral emphasis mask to the audio input signal frames to generate enhanced audio that emphasizes the primary speaker.

A computer-implemented method, comprising:

• receiving an audio input signal including speech from one or more speakers;
• encoding audio input signal frames at a speech-to-text encoder to generate linguistic token data and acoustic speaker embeddings;
• identifying the one or more speakers based on the acoustic speaker embeddings;
• determining, at a context fusion engine, a priority score for each of the one or more speakers;
• selecting, by a focus selector, a primary speaker based on the priority scores;
• generating, by an enhancement engine, a spectral emphasis mask based on at least the selected primary speaker and the acoustic speaker embeddings;
• and
• applying the spectral emphasis mask to the audio input signal frames to generate enhanced audio that emphasizes the primary speaker.