# US 2026/0037744 A1 · Machine-Learned Language Models Which Generate Intermediate Textual Analysis in Service of Contextual Text Generation

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2026/0037744 A1
- Document ID: US20260037744A1
- Status: Published application
- Kind: A1
- Application number: 19/357,854
- Filed: 2025-10-14
- Published: 2026-02-05
- Earliest claimed: 2021-05-21
- Assignees: Google Inc.
- Applicants: Google LLC
- Inventors: Noam Shazeer; Daniel De Freitas Adiwardana
- Web page: https://patentgrep.com/patent/US20260037744A1

## CPC

- G06F 40/35: Discourse or dialogue representation
- G06F 8/38: for implementing user interfaces
- G06F 16/90332: {Natural language query formulation or dialogue systems}
- G06F 16/90335: {Query processing}
- G06F 16/9038: Presentation of query results
- G06F 40/20: Natural language analysis (semantic analysis of natural language G06F40/30)
- G06F 40/279: Recognition of textual entities
- G06F 40/284: Lexical analysis, e.g. tokenisation or collocates
- G06N 3/045: Combinations of networks
- G06N 3/092: Reinforcement learning
- G06N 20/00: Machine learning
- G10L 13/02: Methods for producing synthetic speech; Speech synthesisers
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING

## Abstract

The present disclosure is directed to systems and methods that include and/or leverage one or more machine-learned language models that generate intermediate textual analysis (e.g., including usage of structural tools such as APIs) in service of contextual text generation. For example, a computing system can obtain a contextual text string that includes one or more contextual text tokens. The computing system can process the contextual text string with the machine-learned language model to generate one or more intermediate text strings that include one or more intermediate text tokens. The computing system can process the one or more intermediate text strings with the machine-learned language model to generate an output text string comprising one or more output text tokens. The one or more intermediate text strings can include textual analysis of the contextual text string that supports the output text string.

## Claims

### Claim 1

A computer-implemented method for enabling machine-learned models to invoke programming language operations to improve subsequent model outputs, the method comprising:
obtaining, by a computing system comprising one or more computing devices, an initial sequence;
processing, by the computing system, the initial sequence using a machine-learned model, wherein the machine-learned model uses an attention mechanism to perform attention over the initial sequence;
generating, by the computing system and based on performing attention over the initial sequence, one or more tool tokens associated with a programming language tool;
providing, by the computing system, a tool input for input to the programming language tool;
receiving, by the computing system, a tool response output by the programming language tool, the tool response being based on the tool input;
constructing, by the computing system, an intermediate sequence based on the initial sequence and the tool response;
processing, by the computing system, the intermediate sequence using the machine-learned model, wherein the machine-learned model uses the attention mechanism to perform attention over the intermediate sequence;
generating, by the computing system and based on performing attention over the intermediate sequence, a response sequence; and
outputting, by the computing system, the response sequence.

### Claim 2 (depends on 1)

The computer-implemented method of claim 1, wherein the programming language tool is a programming language interpreter.

### Claim 3 (depends on 2)

The computer-implemented method of claim 2, wherein the programming language interpreter is a Python interpreter.

### Claim 4 (depends on 2)

The computer-implemented method of claim 2, wherein the programming language tool performs a sequence of one or more operations on input data from the initial sequence.

### Claim 5 (depends on 2)

The computer-implemented method of claim 2, comprising:
generating, by the computing system, the initial sequence, wherein generating the initial sequence comprises:
obtaining, by the computing system, an input sequence;
generating, by the computing system and based on performing attention over the input sequence, one or more reasoning tokens that comprise textual analysis of the input sequence; and
constructing, by the computing system, the initial sequence to comprise the input sequence and the reasoning tokens.

### Claim 6 (depends on 5)

The computer-implemented method of claim 5 wherein the textual analysis comprises step-by-step logic for providing a response to the input sequence.

### Claim 7 (depends on 1)

The computer-implemented method of claim 1, wherein the initial sequence comprises contextual text tokens obtained from a user input from a user computing device.

### Claim 8 (depends on 7)

The computer-implemented method of claim 7, wherein the machine-learned model is executed on a server remote from the user computing device.

### Claim 9 (depends on 7)

The computer-implemented method of claim 7, wherein the machine-learned model is executed on the user computing device.

### Claim 10 (depends on 1)

The computer-implemented method of claim 1, wherein the machine-learned model is configured to conduct a dialogue responsive to user inputs.

### Claim 11

A computing system, comprising:
one or more processors; and
one or more non-transitory computer-readable media that collectively store instructions that, when executed by the one or more processors, cause the computing system to perform operations, the operations comprising:
obtaining an initial sequence;
processing the initial sequence using a machine-learned model, wherein the machine-learned model uses an attention mechanism to perform attention over the initial sequence;
generating, based on performing attention over the initial sequence, one or more tool tokens associated with a programming language tool;
providing a tool input for input to the programming language tool;
receiving a tool response output by the programming language tool, the tool response being based on the tool input;
constructing an intermediate sequence based on the initial sequence and the tool response;
processing the intermediate sequence using the machine-learned model, wherein the machine-learned model uses the attention mechanism to perform attention over the intermediate sequence;
generating, based on performing attention over the intermediate sequence, a response sequence; and
outputting the response sequence.

### Claim 12 (depends on 11)

The computing system of claim 11, wherein the programming language tool is a programming language interpreter.

### Claim 13 (depends on 12)

The computing system of claim 12, wherein the programming language interpreter is a Python interpreter.

### Claim 14 (depends on 12)

The computing system of claim 12, wherein the programming language tool performs a sequence of one or more operations on input data from the initial sequence.

### Claim 15 (depends on 12)

The computing system of claim 12, the operations comprising:
generating the initial sequence, wherein generating the initial sequence comprises:
obtaining an input sequence;
generating, based on performing attention over the input sequence, one or more reasoning tokens that comprise textual analysis of the input sequence; and
constructing the initial sequence to comprise the input sequence and the reasoning tokens.

### Claim 16 (depends on 15)

The computing system of claim 15 wherein the textual analysis comprises step-by-step logic for providing a response to the input sequence.

### Claim 17 (depends on 11)

The computing system of claim 11, wherein the initial sequence comprises contextual text tokens obtained from a user input from a user computing device.

### Claim 18 (depends on 17)

The computing system of claim 17, wherein the machine-learned model is executed on a server remote from the user computing device.

### Claim 19 (depends on 17)

The computing system of claim 17, wherein the machine-learned model is executed on the user computing device.

### Claim 20

One or more non-transitory computer-readable media that collectively store instructions that, when executed by one or more processors, cause a computing system to perform operations, the operations comprising:
obtaining an initial sequence;
processing the initial sequence using a machine-learned model, wherein the machine-learned model uses an attention mechanism to perform attention over the initial sequence;
generating, based on performing attention over the initial sequence, one or more tool tokens associated with a programming language tool;
providing a tool input for input to the programming language tool;
receiving a tool response output by the programming language tool, the tool response being based on the tool input;
constructing an intermediate sequence based on the initial sequence and the tool response;
processing the intermediate sequence using the machine-learned model, wherein the machine-learned model uses the attention mechanism to perform attention over the intermediate sequence;
generating, based on performing attention over the intermediate sequence, a response sequence; and
outputting the response sequence.
