Streaming of natural language (NL) based output generated using a large language model (LLM) to reduce latency in rendering thereof

A method implemented by one or more processors, the method comprising:

• receiving natural language (NL) based input associated with a client device;
• and
• generating, based on processing the NL based input using a large language model (LLM), a stream of NL based output that is responsive to the NL based input and that is to be rendered at the client device in a streaming manner, wherein generating the stream of NL based output that is responsive to the NL based input and that is to be rendered at the client device in the streaming manner based on processing the NL based input using the LLM comprises:
• processing, using the LLM, the NL based input to generate a stream of LLM output;
• determining, based on the stream of LLM output, a plurality of first candidate segments for a first segment of the stream of NL based output;
• selecting a given first candidate segment, from among the plurality of first candidate segments for the stream of NL based output, as the first segment of the stream of NL based output;
• and
• in response to selecting the given first candidate segment as the first segment of the stream of NL based output and prior to completion of processing of the NL based input:
• causing the first segment of the stream of NL based output to be rendered at the client device in the streaming manner;
• and
• causing an indication of the first segment of the stream of NL based output to be provided to the LLM to update a state of the LLM;
• and
• continue processing, using the LLM and based on the updated state of the LLM, the NL based input to continue generating the stream of LLM output prior to completion of processing of the NL based input;
• determining, based on the stream of LLM output, a plurality of second candidate segments for a second segment of the stream of NL based output, the second segment of the stream of NL based output following the first segment of the stream of NL based output;
• selecting a given second candidate segment, from among the plurality of second candidate segments for the stream of NL based output, as the second segment of the stream of NL based output;
• and
• in response to selecting the given second candidate segment as the second segment of the stream of NL based output:
• causing the second segment of the stream of NL based output to be rendered at the client device in the streaming manner and subsequent to the first segment of the stream of NL based output.

A system comprising:

• at least one processor;
• and
• memory storing instructions that, when executed, cause the at least one processor to be operable to:
• receive natural language (NL) based input associated with a client device;
• and
• generate, based on processing the NL based input using a large language model (LLM), a stream of NL based output that is responsive to the NL based input and that is to be rendered at the client device in a streaming manner, wherein the instructions to generate the stream of NL based output that is responsive to the NL based input and that is to be rendered at the client device in the streaming manner based on processing the NL based input using the LLM comprise instructions to:
• process, using the LLM, the NL based input to generate a stream of LLM output;
• determine, based on the stream of LLM output, a plurality of first candidate segments for a first segment of the stream of NL based output;
• select a given first candidate segment, from among the plurality of first candidate segments for the stream of NL based output, as the first segment of the stream of NL based output;
• and
• in response to selecting the given first candidate segment as the first segment of the stream of NL based output and prior to completion of processing of the NL based input:
• cause the first segment of the stream of NL based output to be rendered at the client device in the streaming manner;
• and
• cause an indication of the first segment of the stream of NL based output to be provided to the LLM to update a state of the LLM;
• and
• continue to process, using the LLM and based on the updated state of the LLM, the NL based input to continue generating the stream of LLM output prior to completion of processing of the NL based input;
• determine, based on the stream of LLM output, a plurality of second candidate segments for a second segment of the stream of NL based output, the second segment of the stream of NL based output following the first segment of the stream of NL based output;
• select a given second candidate segment, from among the plurality of second candidate segments for the stream of NL based output, as the second segment of the stream of NL based output;
• and
• in response to selecting the given second candidate segment as the second segment of the stream of NL based output:
• cause the second segment of the stream of NL based output to be rendered at the client device in the streaming manner and subsequent to the first segment of the stream of NL based output.