# US 2026/0259893 A1 · Query clarification based on confidence in a classification performed by a generative language machine learning model

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2026/0259893 A1
- Document ID: US20260259893A1
- Status: Published application
- Kind: A1
- Application number: 19/653,960
- Filed: 2026-04-21
- Published: 2026-09-03
- Earliest claimed: 2024-05-22
- Assignees: Shopify Inc.
- Applicants: Shopify Inc.
- Inventors: Vesna Brown; Maxwell Dayvson Da Silva; David Wurtz; Alexandre Salle; Christopher Bullock
- Web page: https://patentgrep.com/patent/US20260259893A1

## CPC

- G06F 16/285: {Clustering or classification}
- G06F 16/2455: Query execution
- G06F 16/3329: Natural language query formulation
- G06F 16/3346: {using probabilistic model}
- G06F 16/353: into predefined classes
- G06F 30/27: using machine learning, e.g. artificial intelligence, neural networks, support vector machines [SVM] or training a model
- G06N 3/08: Learning methods
- G06F 40/10: Text processing (natural language analysis G06F40/20; semantic analysis G06F40/30; processing or translation of natural language G06F40/40)
- G06F 40/216: using statistical methods
- G10L 15/18: using natural language modelling
- G10L 15/183: using context dependencies, e.g. language models
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING

## Abstract

A large language model (LLM) may be used to classify an input into one of a plurality of categories. However, given the machine-learning operation of the LLM, the output of the LLM does not represent a definitive statement, but is based on probability computations of the machine learning model. Therefore, the classification performed by the LLM might not be correct. Classification into the wrong category by the LLM results in downstream technical problems. In some implementations, when an LLM generates a response that classifies an input, one or more probability values associated with a token that forms the basis of the response may be used to determine a confidence value. The confidence value is indicative of confidence in the classification performed by the LLM. An action may be taken based on the confidence value.

## Claims

### Claim 1

A computer-implemented method comprising:
receiving, from a generative language model, output indicating a query is unambiguous and a probability value associated with a token that forms a basis of the output indicating the query is unambiguous;
determining a confidence value using at least the probability value associated with the token that forms the basis of the output indicating the query is unambiguous; and
taking an action responsive to the confidence value not satisfying a threshold.

### Claim 2 (depends on 1)

The computer-implemented method of claim 1, wherein the confidence value is indicative of confidence in classification of the query as unambiguous.

### Claim 3 (depends on 1)

The computer-implemented method of claim 1, wherein the action comprises providing a reply indicating that the query is ambiguous.

### Claim 4 (depends on 1)

The computer-implemented method of claim 1, wherein the action comprises providing a reply that requests clarification relating to the query.

### Claim 5 (depends on 1)

The computer-implemented method of claim 1, wherein the action comprises stopping the generative language model from generating further output.

### Claim 6 (depends on 1)

The computer-implemented method of claim 1, wherein the output is a first output, and wherein the action comprises:
modifying the first output to indicate the query is ambiguous;
continuing generation of second output from the generative language model with the first output modified; and
providing a reply based on the second output from the generative language model.

### Claim 7 (depends on 1)

The computer-implemented method of claim 1, wherein the token that forms the basis of the output indicating the query is unambiguous is a selected token, wherein the method further comprises receiving, from the generative language model, a probability value associated with an alternative token that was not selected by the generative language model to form the basis of the output indicating the query is unambiguous, and wherein the confidence value is determined using the probability value associated with the selected token and the probability value associated with the alternative token.

### Claim 8 (depends on 7)

The computer-implemented method of claim 7, wherein the alternative token that was not selected forms a basis of language indicating the query is ambiguous.

### Claim 9 (depends on 7)

The computer-implemented method of claim 7, wherein the alternative token that was not selected has a next highest associated probability after the selected token.

### Claim 10 (depends on 7)

The computer-implemented method of claim 7, wherein the query can be classified into one of only two categories, the two categories being a first category corresponding to the query being unambiguous and a second category corresponding to the query being ambiguous, wherein a response classifying the query can be one of two response values, wherein a first response value of the two response values corresponds to the query being unambiguous and the second response value of the two response values corresponds to the query being ambiguous, wherein the response is the first response value corresponding to the query being unambiguous, wherein the selected token corresponds to the first response value, and wherein the alternative token corresponds to the second response value.

### Claim 11

A system comprising:
at least one processor; and
a memory storing processor-executable instructions that, when executed by the at least one processor, cause the system to:
receive, from a generative language model, output indicating a query is unambiguous and a probability value associated with a token that forms a basis of the output indicating the query is unambiguous;
determine a confidence value using at least the probability value associated with the token that forms the basis of the output indicating the query is unambiguous; and
take an action responsive to the confidence value not satisfying a threshold.

### Claim 12 (depends on 11)

The system of claim 11, wherein the confidence value is indicative of confidence in classification of the query as unambiguous.

### Claim 13 (depends on 11)

The system of claim 11, wherein the action comprises providing a reply indicating that the query is ambiguous.

### Claim 14 (depends on 11)

The system of claim 11, wherein the action comprises providing a reply that requests clarification relating to the query.

### Claim 15 (depends on 11)

The system of claim 11, wherein the action comprises stopping the generative language model from generating further output.

### Claim 16 (depends on 11)

The system of claim 11, wherein the output is a first output, and wherein the action comprises:
modifying the first output to indicate the query is ambiguous;
continuing generation of second output from the generative language model with the first output modified; and
providing a reply based on the second output from the generative language model.

### Claim 17 (depends on 11)

The system of claim 11, wherein the token that forms the basis of the output indicating the query is unambiguous is a selected token, wherein the instructions, when executed, further cause the system to receive, from the generative language model, a probability value associated with an alternative token that was not selected by the generative language model to form the basis of the output indicating the query is unambiguous, and wherein the confidence value is determined using the probability value associated with the selected token and the probability value associated with the alternative token.

### Claim 18 (depends on 17)

The system of claim 17, wherein the alternative token that was not selected forms a basis of language indicating the query is ambiguous.

### Claim 19 (depends on 17)

The system of claim 17, wherein the alternative token that was not selected has a next highest associated probability after the selected token.

### Claim 20

A non-transitory computer readable medium having stored thereon computer-executable instructions that, when executed by a computer, cause the computer to perform operations comprising:
receiving, from a generative language model, output indicating a query is unambiguous and a probability value associated with a token that forms a basis of the output indicating the query is unambiguous;
determining a confidence value using at least the probability value associated with the token that forms the basis of the output indicating the query is unambiguous; and
taking an action responsive to the confidence value not satisfying a threshold.
