Device State Aware Dynamic Automatic Speech Recognition Optimization

A system comprising:

• data processing hardware;
• and
• memory hardware in communication with the data processing hardware, the memory hardware storing instructions that when executed on the data processing hardware cause the data processing hardware to perform operations comprising:
• receiving a text-to-speech (TTS) end event indicating that audible output of first TTS audio from a user device associated with a user is finished, the first TTS audio characterizing a first response generated by a large language model (LLM)-powered assistant that is directed toward the user during a voice-based conversation between the user and the LLM-powered assistant;
• based on receiving the TTS end event, instructing an automated speech recognition (ASR) system to use a first level of ASR processing for performing speech recognition on anticipated user speech;
• as the user speaks a natural language query that solicits a second response from the LLM-powered assistant during the voice-based conversation:
• receiving an audio data characterizing the natural language query, the audio data captured by the microphone in communication with the user device;
• and
• performing, by the ASR system, using the first level of ASR processing, speech recognition on the audio data to generate a speech recognition result for the natural language query;
• processing, by the LLM-powered assistant, the speech recognition result for the natural language query to generate the second response solicited by the natural language query spoken by the user;
• receiving a TTS start event indicating that second TTS audio is about to be audibly output from the user device, the second TTS audio characterizing the second response generated by the LLM-powered assistant;
• and
• based on receiving the TTS start event, instructing the ASR system to use a second level of ASR processing for performing speech recognition of any user speech captured by the microphone while the second TTS audio is being audibly output from the user device, the second level of ASR processing different than the first level of ASR processing.