# US 2026/0064970 A1 · Entropy-based detection of the fluency of machine-generated text

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2026/0064970 A1
- Document ID: US20260064970A1
- Status: Published application
- Kind: A1
- Application number: 18/826,012
- Filed: 2024-09-05
- Published: 2026-03-05
- Earliest claimed: 2024-09-05
- Assignees: Microsoft Technology Licensing, LLC
- Applicants: Microsoft Technology Licensing, LLC.
- Inventors: SAMER AAMAR; ABEDELKADER ASI; ROY EISENSTADT; SHAHAR KEREN; ALEXANDER TSVETKOV
- Web page: https://patentgrep.com/patent/US20260064970A1

## CPC

- G06F 40/216: using statistical methods
- G06F 40/284: Lexical analysis, e.g. tokenisation or collocates
- G06F 40/253: Grammatical analysis; Style critique
- G06F 40/30: Semantic analysis
- G06F 40/35: Discourse or dialogue representation
- G06F 40/44: Statistical methods, e.g. probability models
- G06F 40/56: Natural language generation
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

An entropy-based technique is used to select a large language model capable of generating fluent natural language text. An entropy model, trained on fluent natural language samples, is used to determine the entropy of a large language model based on an output text generated by the large language model. The entropy of a machine-generated natural language text is used to quantify the amount of information that the large language model holds with respect to the tokens and context of an input text segment. The entropy score of a model is then used to select a large language model capable of generating fluent text or to select the most fluent machine-generated output text produced by a set of large language models.

## Claims

### Claim 1

A system, comprising:
a processor; and
a memory that stores a program that is configured to be executed by the processor, the program comprises instructions to perform actions that:
obtain an entropy model trained on fluent natural language text;
invoke a plurality of large language models to perform a task that generates an output natural language text given an input natural language text, wherein the output natural language text comprises an ordered sequence of tokens;
invoke the entropy model with each of the output natural language text generated by each of the plurality of large language models, wherein the entropy model generates an output probability for each token in a respective output natural language text;
compute an entropy score for each of the plurality of large language models, wherein the entropy score for a select large language model is based on output probabilities generated by the entropy model given an output natural language text generated by the select large language model; and
upon a select one of the plurality of large language models having a low entropy score, deploy the selected large language model to generate fluent natural language text for a given input text.

### Claim 2 (depends on 1)

The system of claim 1, wherein the program comprises instructions to perform actions that:
upon a select one of the plurality of large language models having a low entropy score, output the output natural language text of the select one of the plurality of large language models as being fluent natural language text.

### Claim 3 (depends on 1)

The system of claim 1, wherein the program comprises instructions to perform actions that:
construct a training dataset of fluent natural language text; and
pre-train a large language model with the training dataset using a mask language modeling objective to produce the entropy model.

### Claim 4 (depends on 1)

The system of claim 1, wherein the program comprises instructions to perform actions that:
compute the entropy score as a sum of each probability of each token in the output text.

### Claim 5 (depends on 1)

The system of claim 1, wherein the fluent natural language text comprises non-vulgar language.

### Claim 6 (depends on 1)

The system of claim 1, wherein the input natural language text comprises a call transcript, wherein the output natural language text comprises an email responding to the call transcript.

### Claim 7 (depends on 1)

The system of claim 1, wherein the plurality of large language models comprises at least one neural transformer model with attention.

### Claim 8 (depends on 1)

The system of claim 1, wherein the entropy model is a neural transformer model with attention.

### Claim 9

A computer-implemented method, comprising:
accessing an entropy model trained on fluent natural language text;
invoking at least one large language model to generate an output natural language text for a given an input natural language text, wherein the output natural language text comprises an ordered sequence of tokens;
invoking the entropy model with the output natural language text, wherein the entropy model generates a conditional output probability for each token in the output natural language text;
determining an entropy score for the at least one large language model by accumulating the conditional output probability generated by the entropy model for each token in the output natural language text; and
upon the entropy score indicating low entropy, outputting the output natural language text as fluent natural language.

### Claim 10 (depends on 9)

The computer-implemented method of claim 9, further comprising:
upon the entropy score indicating high entropy, discarding the output natural language text.

### Claim 11 (depends on 9)

The computer-implemented method of claim 9, wherein the entropy score comprises a sum of each probability generated by the entropy model for each token in the output natural language text.

### Claim 12 (depends on 9)

The computer-implemented method of claim 9,
wherein the input natural language text is a call transcript, and
wherein the output natural language text is an email pertaining to the call transcript.

### Claim 13 (depends on 12)

The computer-implemented method of claim 12, further comprising:
transmitting the email to a caller of the call transcript.

### Claim 14 (depends on 9)

The computer-implemented method of claim 9, wherein the fluent natural language text comprises non-vulgar natural language.

### Claim 15 (depends on 9)

The computer-implemented method of claim 9, wherein the entropy model is a neural transformer model with attention.

### Claim 16

A hardware storage device having stored thereon computer executable instructions that are structured to be executable by a processor of a computing device to thereby cause the computing device to perform actions that:
obtain an entropy model configured to recognize fluent natural language text;
invoke a large language model to generate an output natural language text for an input natural language text, wherein the output natural language text comprises an ordered sequence of tokens;
generate an output probability for each token in the output natural language text from the entropy model, wherein the entropy model is given the output natural language text, wherein the output probability for each token represents a likelihood of a select token in the output natural language text following previous tokens in the output natural language text;
accumulate the output probabilities of each token in the output natural language text generated by the entropy model, wherein the accumulated output probabilities represent an entropy of the large language model; and
when the entropy of the large language model meets a threshold, output the output natural language text as fluent and deploy the large language model to generate fluent natural language text for a target application.

### Claim 17 (depends on 16)

The hardware storage device of claim 16 having stored thereon computer executable instructions that are structured to be executable by a processor of a computing device to thereby cause the computing device to perform actions that:
when the entropy of the large language model fails to meet a threshold, discard the output natural language text.

### Claim 18 (depends on 16)

The hardware storage device of claim 16 having stored thereon computer executable instructions that are structured to be executable by a processor of a computing device to thereby cause the computing device to perform actions that:
pre-train the entropy model with a training dataset comprising fluent natural language samples.

### Claim 19 (depends on 16)

The hardware storage device of claim 16, wherein the entropy of the large language model comprises a sum of each probability of each token in the output natural language text generated by the entropy model.

### Claim 20 (depends on 16)

The hardware storage device of claim 16, wherein the entropy model is a neural transformer model with attention.
