Mixing Heterogeneous Loss Types to Improve Accuracy of Keyword Spotting

A system comprising:

• data processing hardware;
• and
• memory hardware in communication with the data processing hardware, the memory hardware storing instructions that when executed on the data processing hardware cause the data processing hardware to perform operations comprising:
• receiving a training input audio sequence comprising a sequence of input frames, the sequence of input frames defining a hotword that initiates a wake-up process on a user device;
• obtaining a first label for the training input audio sequence;
• obtaining a second label for the training input audio sequence, the second label different than the first label;
• generating, using a memorized neural network and the training input audio sequence, an output indicating a likelihood the training input audio sequence includes the hotword;
• determining a first loss based on the first label and the output;
• determining a second loss based on the second label and the output;
• and
• optimizing the memorized neural network based on the first loss and the second loss associated with the training input audio sequence.