Query response interface with server side generative model(s)

A system comprising one or more processors and memory storing instructions that, in response to execution by one or more of the processors, cause one or more of the processors to:

• establish a persistent bidirectional network connection with a remote client device;
• receive, via the persistent bidirectional network connection, a sequence of instances of user query data representing a user voice query, wherein one or more instances of the sequence are received at the server computing system before the user voice query is completely captured at the remote client device;
• process each received instance of the sequence using a generative model at the server computing system, wherein the processing comprises dynamically updating a server-side buffer and caching input tokens based on the received instances;
• while continuing to receive subsequent instances of the user query data via the persistent bidirectional network connection, generate streaming output responsive to the user voice query using the generative model;
• and
• transmit one or more portions of the streaming output to the remote client device via the persistent bidirectional network connection as the streaming output is generated.

At least one non-transitory computer-readable medium comprising instructions that, in response to execution of the instructions by one or more processors, cause the one or more processors to:

• establish a persistent bidirectional network connection with a remote client device;
• receive, via the persistent bidirectional network connection, a sequence of instances of user query data representing a user voice query, wherein one or more instances of the sequence are received at the server computing system before the user voice query is completely captured at the remote client device;
• process each received instance of the sequence using a generative model at the server computing system, wherein the processing comprises dynamically updating a server-side buffer and caching input tokens based on the received instances;
• while continuing to receive subsequent instances of the user query data via the persistent bidirectional network connection, generate streaming output responsive to the user voice query using the generative model;
• and
• transmit one or more portions of the streaming output to the remote client device via the persistent bidirectional network connection as the streaming output is generated.