# US 2025/0148217 A1 · Hybrid inference for an efficient, low latency llm-based assistant

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2025/0148217 A1
- Document ID: US20250148217A1
- Status: Published application
- Kind: A1
- Application number: 18/387,768
- Filed: 2023-11-07
- Published: 2025-05-08
- Earliest claimed: 2023-11-07
- Assignees: Google Inc.
- Applicants: Google LLC
- Inventors: Matthew Sharifi; Victor Carbune
- Web page: https://patentgrep.com/patent/US20250148217A1

## CPC

- G06F 40/40: Processing or translation of natural language (natural language analysis G06F40/20; semantic analysis G06F40/30)
- G06F 40/56: Natural language generation
- G06F 40/166: Editing, e.g. inserting or deleting
- G06F 40/30: Semantic analysis
- G06N 3/045: Combinations of networks
- G06N 3/0475: Generative networks
- G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue
- G10L 2015/223: {Execution procedure of a spoken command}
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING

## Abstract

Implementations utilize a hybrid use of a smaller LLM and a larger LLM to generate and refine content responsive to a user query/request for content generation. In various implementations, the smaller LLM is utilized to process the user query for content generation, to generate initial content responsive to the user query for content generation. The user query for content generation and the initial content can be utilized to generate a text prompt, where the text prompt can be configured to further include a request for focused edit(s). Such a text prompt can be processed using the larger LLM, to generate focused edit(s) to the initial content that refine the initiated content, so that revised content (with improved accuracy) responsive to the user query for content generation is acquired.

## Claims

### Claim 1

A method implemented by one or more processors, the method comprising:
receiving, via a client device, a user query for content generation
in response to receiving the user query for content generation,
processing the user query using a first LLM, to generate a first LLM output,
causing initial content, that is based on the first LLM output to be rendered via a user interface of the client device;
generating a text prompt based on the first LLM output, the text prompt further including a request for one or more focused edits;
providing the text prompt to a second LLM that is less computationally efficient than the first LLM,
receiving, in response to providing the text prompt, one or more focused edits to replace one or more initial segments in the initial content with one or more updated segments,
wherein the one or more focused edits are generated using the second LLM to perform one or more iterations of content refinement based on the text prompt; and
causing the one or more focused edits to the initial content to be visually rendered via the user interface, resulting in revised content responsive to the user query for content generation.

### Claim 2 (depends on 1)

The method of claim 1, wherein the one or more initial segments each corresponds to a sentence, and wherein causing the one or more focused edits to the initial version of the document to be visually rendered via the user interface comprises:
causing a first focused edit to the initial content to be visually rendered via the user interface, wherein the first focused edit is generated during a first iteration of content refinement and replaces a first initial sentence in the initial content with a first updated sentence, and
causing a second focused edit to be visually rendered via the user interface, wherein the second focused edit is generated during a second iteration of content refinement and replaces a second initial sentence in the initial content with a second updated sentence, the second initial sentence being different from the first initial sentence.

### Claim 3 (depends on 2)

The method of claim 2, wherein the text prompt is processed as input using the second LLM during the first iteration to generate the first focused edit, and wherein a second text prompt different from the text prompt is processed as input using the second LLM during the second iteration to generate the second focused edit.

### Claim 4 (depends on 3)

The method of claim 3, wherein the second text prompt corresponds to the text prompt incorporating the first focused edit to the initial content.

### Claim 5 (depends on 1)

The method of claim 1, further comprising:
receiving, via the user interface, a first user edit while the one or more focused edits are being applied to the initial content; and
causing the first user edit to be applied to the initial document while the one or more focused edits are being applied to the initial content.

### Claim 6 (depends on 1)

The method of claim 1, further comprising:
receiving, via the user interface, a second user edit to the revised content; and
causing the second user edit to be applied to the revised content.

### Claim 7 (depends on 1)

The method of claim 1, wherein the request for one or more focused edits is a request for replacing a single sentence or a single paragraph in the initial content during each iteration of the one or more iterations.

### Claim 8 (depends on 1)

The method of claim 1, wherein the text prompt includes the initial content generated based on the first LLM output.

### Claim 9 (depends on 1)

The method of claim 1, wherein the text prompt further includes a particular sentence refinement request to refine a particular initial sentence in the initial content.

### Claim 10 (depends on 9)

The method of claim 9, wherein the particular sentence refinement request is generated based on the first LLM output indicating the particular sentence to be refined.

### Claim 11 (depends on 1)

The method of claim 1, wherein generating the text prompt based on the user query and the initial content comprises:
filtering the initial content to redact privacy information from the initial content, thereby generating a redacted version of the initial content; and
generating the text prompt to include the redacted version of the initial content.

### Claim 12 (depends on 1)

The method of claim 1, further comprising:
generating a training instance for the first LLM, wherein the training instance includes:
the user query for content generation as a training instance input, and
the revised content as a ground truth output.

### Claim 13 (depends on 12)

The method of claim 12, further comprising:
training the first LLM using the generated training instance, including:
processing the user query for content generation as input using the first LLM, to generate an output of the first LLM,
comparing the output of the first LLM with the ground truth output, and
updating one or more weights of the first LLM based on comparing the output of the first LLM with the ground truth output.

### Claim 14

A method implemented by one or more processors, the method comprising:
receiving a text prompt generated based on a user query for content generation and initial content responsive to the user query for content generation, the text prompt further including a request for one or more focused edits,
wherein the user query is received via a client device, and
wherein the initial content is visually rendered via a user interface of the client device;
determining, based on the text prompt and using a LLM to perform one or more iterations of content refinement, one or more focused edits to the initial content; and
providing the one or more focused edits or a subset of the one or more focused edits, the providing causes the one or more focused edits or the subset to be rendered at the user interface of the client device.

### Claim 15 (depends on 14)

The method of claim 14, wherein the initial content responsive to the user query for content generation is generated based on processing of the user query for content generation using an additional LLM, the additional LLM having a less quantity of parameters than the LLM.

### Claim 16 (depends on 14)

The method of claim 14, wherein the request for one or more focused edits is to replace a single sentence in the initial content with an updated sentence.

### Claim 17 (depends on 14)

The method of claim 14, wherein providing the one or more focused edits or the subset comprises:
providing a first focused edit to the initial content at completion of a first iteration of content refinement and prior to completion of a second iteration of content refinement, and
providing a second focused edit different from the first focused edit at completion of the second iteration of content refinement.

### Claim 18 (depends on 14)

The method of claim 14, wherein the text prompt includes the initial content generated based on the first LLM output that corresponds to the user query for content generation, in addition to the request for one or more focused edits.

### Claim 19 (depends on 14)

The method of claim 14, wherein the initial content is modified to exclude privacy information, and the text prompt includes the modified initial content and the request for one or more focused edits.

### Claim 20

A method implemented by one or more processors, the method comprising:
receiving, via a client device, a spoken user query for content generation
in response to receiving the spoken user query for content generation,
processing the spoken user query to generate a speech recognition of the spoken user query in natural language,
processing the speech recognition of the spoken user query using a first LLM, to generate a first LLM output, and
causing initial content that is based on the first LLM output to be rendered via a user interface of the client device;
generating a text prompt that includes the speech recognition of the user query for content generation, the initial content generated based on the first LLM output, and a request for one or more focused edits;
providing the text prompt to a second LLM that is less computationally efficient than the first LLM,
receiving, in response to providing the text prompt, one or more focused edits to replace one or more initial segments in the initial content with one or more updated segments, the one or more focused edits being generated based on processing of the text prompt using the second LLM for one or more iterations of content refinement, and
causing the one or more focused edits to the initial content to be visually rendered via the user interface, resulting in revised content responsive to the spoken user query for content generation.
