Incremental context composition for latency reduction in language processing machine learning models

A method for reducing latency in generating responses using a large language model (LLM), comprising:

• applying a sentence boundary detection technique to an input buffer in order to identify a first sentence in a stream of input text while the stream of input text is being input to the input buffer via a user interface;
• performing a forward pass through an encoder portion of the LLM to generate a first embedding of the first sentence;
• determining that a final sentence has been entered in the stream of input text;
• performing an additional forward pass through the encoder portion of the LLM to generate a second embedding of the final sentence;
• generating a cumulative context vector using a merging machine learning model based on the first embedding and the second embedding;
• and
• injecting the cumulative context vector into a decoder portion of the LLM, bypassing redundant re-encoding of the first sentence and the final sentence;
• and
• receiving a response to the stream of input text as an output from the LLM based on the cumulative context vector.

A non-transitory computer readable medium comprising instructions that, when executed by one or more processors of a computing system, cause the computing system to:

• apply a sentence boundary detection technique to an input buffer in order to identify a first sentence in a stream of input text while the stream of input text is being input to the input buffer via a user interface;
• perform a forward pass through an encoder portion of a large language model (LLM) to generate a first embedding of the first sentence;
• determine that a final sentence has been entered in the stream of input text;
• perform an additional forward pass through the encoder portion of the LLM to generate a second embedding of the final sentence;
• generate a cumulative context vector using a merging machine learning model based on the first embedding and the second embedding;
• and
• inject the cumulative context vector into a decoder portion of the LLM, bypassing redundant re-encoding of the first sentence and the final sentence;
• and
• receive a response to the stream of input text as an output from the LLM based on the cumulative context vector.