Query response interface with server side generative model(s)

A method implemented by one or more processors, the method comprising:

• identifying an instance of audio data capturing a user voice query captured via one or more microphones of a client device;
• generating, at the client device, a sequence of instances of tokenizable query text by processing the instance of audio data using an automatic speech recognition model, where the sequence of instances of tokenizable query text are a text representation of the user voice query;
• for each of the instances, and in the sequence:
• transmitting the instance of tokenizable query text to a computing system remote from the client device, where one or more of the instances are transmitted before generation of the entire sequence has completed;
• receiving, from the computing system, streaming output responsive to the user voice query, wherein the streaming output responsive to the user voice query is generated at the computing system by processing the sequence of instances of tokenizable query text using a generative model (GM) which includes streamed multi-head attention to generate the streaming output;
• and
• rendering output based on the streaming output responsive to the user voice query via one or more user interface output devices of the client device.

A method implemented by one or more processors, the method comprising:

• receiving, at a computing system remote from a client device, a sequence of instances of tokenizable query text,
• wherein the sequence of instances of tokenizable query text are a text representation of a user voice query generated at the client device, where the sequence is generated by processing audio data capturing the user voice query using an automatic speech recognition model, and
• wherein one or more instances of the sequence are transmitted before generation of the entire sequence at the client device has completed;
• for each instance, and in the sequence:
• processing the instance using a generative model (GM), wherein processing the instance using the GM comprises:
• updating one or more portions of an input layer of the GM based on the instance;
• generating streamed multi-head attention output based on processing the updated input layer using the GM;
• while continuing to receive subsequent instances in the sequence, generating streaming output responsive to the user voice query based on the processing the multi-head attention output using the GM;
• and
• transmitting one or more portions of the streaming output responsive to the user voice query to the client device.