US 2026/0099528 A1 · LLM LATENCY REDUCTION VIA BRIDGING MULTIPLE LLMS OF DIFFERING SIZES

Abstract

Implementations utilize a smaller LLM to generate content responsive to a user query and cause a portion of the generated content to be rendered as an immediate response to the user query. Implementations further utilize a larger LLM to generate content that starts with the portion of the generated content and that includes a refined portion succeeding the portion of the generated content. The refined portion can be rendered succeeding the portion of the generated content. In some implementations, instead of using the smaller LLM, alternatively, the portion of the generated content rendered as the immediate response can be generated based on a default text string or a template, where the template can be determined/selected from a plurality of predefined templates based on a natural language understanding of the user query.

Claims20

Dependent claims

A system comprising:

memory storing instructions;
and
one or more processors operable to execute the instructions to:
receive a user query that includes natural language, the user query being generated based on user interface input;
and
in response to receiving the user query:
process the user query, using a first generative model, to generate a natural language response that is responsive to the user query;
cause at least a first portion of the natural language response to be rendered;
generate a prompt that includes the user query and the first portion of the natural language response;
cause the generated prompt to be processed by a second generative model, to generate a refined natural language response that is responsive to the user query and that includes a refined portion to follow the first portion;
and
cause the refined portion of the refined natural language response to be rendered.

9 more elements

Dependencies are inferred from the claim text.

Llm latency reduction via bridging multiple llms of differing sizes

Life of this patent

Llm latency reduction via bridging multiple llms of differing sizes

Life of this patent

Abstract

Claims20