Speaker diartzation using an end-to-end model

A method implemented by one or more processors, the method comprising:

• receiving a stream of audio data that is based on output from one or more microphones of a client device, the client device including an automated assistant interface for an automated assistant, and the stream of audio data received in response to invocation of the automated assistant;
• generating a sequence of audio data frames as the stream of audio data is received;
• for each of the audio data frames, and in the sequence:
• applying frame features for the audio data frame as input to a trained recurrent neural network (RNN) model;
• processing the frame features for the audio data frame using the trained RN N model to generate direct output, of the trained RNN model, that includes a corresponding probability for each of a plurality of permutation invariant speaker labels, wherein the trained RNN model comprises:
• a long short-term memory (LSTM) layer, and
• an affine layer as a final layer, the affine layer having an output dimension that conforms to the plurality of speaker labels;
• and
• for each of a first plurality of the audio data frames, assigning a corresponding one of the plurality of speaker labels to the audio data frame in response to the corresponding probability, for the audio data frame, satisfying a threshold;
• for each of a second plurality of the audio data frames, assigning an unknown label to the audio data frame in response to the corresponding probabilities, generated as direct output of the trained RNN model for the audio data frame, all failing to satisfy the threshold;
• and
• using, by the automated assistant, the assigned speaker labels and the assigned unknown labels in processing of the stream of audio data.

A client device, comprising:

• one or more microphones;
• an automated assistant interface for an automated assistant;
• memory storing instructions;
• one or more processors operable to execute the instructions to:
• receive a stream of audio data that is based on output from the one or more microphones;
• generate a sequence of audio data frames as the stream of audio data is received;
• for each of the audio data frames, and in the sequence:
• apply frame features for the audio data frame as input to a trained recurrent neural network (RNN) model;
• process the frame features for the audio data frame using the trained RNN model to generate direct output, of the trained RNN model, that includes a corresponding probability for each of a plurality of permutation invariant speaker labels, wherein the trained RNN model comprises:
• a long short-term memory (LSTM) layer, and
• an affine layer as a final layer, the affine layer having an output dimension that conforms to the plurality of speaker labels;
• and
• for each of a first plurality of the audio data frames, assign a corresponding one of the plurality of speaker labels to the audio data frame in response to the corresponding-probability, for the audio data frame, satisfying a threshold;
• for each of a second plurality of the audio data frames, assign an unknown label to the audio data frame in response to the corresponding probabilities, generated as direct output of the trained RNN model for the audio data frame, all failing to satisfy the threshold;
• and
• use, by the automated assistant, the assigned speaker labels and the assigned unknown labels in processing of the stream of audio data.