Neural networks for speaker verification

A system comprising:

• data processing hardware of a user device;
• and
• memory hardware of the user device and in communication with the data processing hardware, the memory hardware storing instructions that when executed by the data processing hardware cause the data processing hardware to perform operations comprising:
• prompting a user of the user device to speak a plurality of enrollment utterances;
• receiving audio signals that represent the user speaking the plurality of enrollment utterances, the audio signals representing the user speaking the plurality of enrollment utterances recorded by the user device;
• generating, using a trained neural network, a reference speaker model based on the audio signals that represent the user speaking the plurality of enrollment utterances, the reference speaker model associated with the user and characterizing distinctive features of a voice of the user, the trained neural network comprising:
• a long short-term memory (LSTM) layer configured to receive, as input, the audio signals representing the user speaking the plurality of enrollment utterances and generate an enrollment LSTM output;
• and
• a fully-connected linear layer configured to receive, as input, the enrollment LSTM output and generate, as output, the reference speaker model, wherein the fully-connected linear layer comprises a last layer of the trained neural network;
• storing the reference speaker model on memory hardware of the user device;
• obtaining a plurality of audio frames representing a text-independent utterance comprising a different utterance from each of the plurality of enrollment utterances;
• generating, using the trained neural network, a speaker representation for the text-independent utterance, the speaker representation indicating distinctive features of a voice of a speaker of the text-independent utterance, wherein the speaker representation is generated as output from the fully-connected linear layer of the trained neural network;
• determining that a similarity score between the speaker representation for the text-independent utterance and the reference speaker model stored on the memory hardware of the user device satisfies a similarity score threshold;
• and
• based on determining that the similarity score satisfies the similarity score threshold, authenticating the speaker of the text-independent utterance as the user associated with the reference speaker model.