Enabling natural conversations with soft endpointing for an automated assistant

A method implemented by one or more processors, the method comprising:

• processing, using an automatic speech recognition (ASR) model, a stream of audio data to generate a stream of ASR output, the stream of audio data being generated by one or more microphones of the client device, and the stream of audio data capturing a portion of a spoken utterance of the user that is directed to an automated assistant implemented at least in part at the client device;
• processing, using a natural language understanding (NLU) model, the stream of ASR output, to generate a stream of NLU output;
• determining, based on at least the stream of NLU output, whether the user has paused in providing the spoken utterance or has completed providing the spoken utterance;
• and
• in response to determining that the user has paused in providing the spoken utterance and has not completed providing of the spoken utterance:
• determining natural conversation output to be provided for audible presentation to the user, the natural conversation output to be provided for audible presentation to the user to indicate the automated assistant is waiting for the user to complete providing of the spoken utterance;
• and
• causing the natural conversation output to be provided for audible presentation to the user via one or more speakers of the client device.

A method implemented by one or more processors, the method comprising:

• processing, using an automatic speech recognition (ASR) model, a stream of audio data to generate a stream of ASR output, the stream of audio data being generated by one or more microphones of the client device, and the stream of audio data capturing a portion of a spoken utterance of the user that is directed to an automated assistant implemented at least in part at the client device;
• processing, using a natural language understanding (NLU) model, the stream of ASR output, to generate a stream of NLU output;
• determining whether the user has paused in providing the spoken utterance or has completed providing the spoken utterance;
• and
• in response to determining that the user has paused in providing the spoken utterance and has not completed providing of the spoken utterance:
• determining natural conversation output to be provided for audible presentation to the user, the natural conversation output to be provided for audible presentation to the user to indicate the automated assistant is waiting for the user to complete providing of the spoken utterance;
• causing the natural conversation output to be provided for audible presentation to the user via one or more speakers of the client device;
• and
• in response to determining the user has not completed providing of the spoken utterance within a threshold duration of time subsequent to causing the natural conversation output to be provided for audible presentation to the user via the one or more speakers of the client device:
• determining, based on at least the stream of NLU data, whether the automated assistant can initiate fulfillment of the spoken utterance;
• and
• in response to determining that the automated assistant can initiate fulfillment of the spoken utterance based on the stream of NLU data:
• causing the automated assistant to initiate fulfillment of the spoken utterance.