Speech recognition with selective use of dynamic language models

A system comprising:

• data processing hardware;
• and
• memory hardware in communication with the data processing hardware and storing instructions that when executed on the data processing hardware cause the data processing hardware to perform operations comprising
• receiving an audio signal characterizing an utterance captured by a computing device associated with a user;
• generating, using a class-based language model, a word lattice representing a transcription hypothesis path corresponding to a respective transcription for the utterance, the transcription hypothesis path including a class identifier;
• obtaining, based on the word lattice, a list of user-specific class-based terms, each user-specific class-based term in the list of user-specific class-based terms belonging to a class associated with the class identifier included in the transcription hypothesis path;
• modifying the word lattice to include the list of user-specific class-based terms;
• re-training the class-based language model using the modified word lattice;
• and
• providing, using the re-trained class-based language model, a speech recognition result for the utterance.