# US 2026/0064745 A1 · Out-of-distribution query detection for improving language model generation

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2026/0064745 A1
- Document ID: US20260064745A1
- Status: Published application
- Kind: A1
- Application number: 18/818,569
- Filed: 2024-08-28
- Published: 2026-03-05
- Earliest claimed: 2024-08-28
- Assignees: Intuit Inc.
- Applicants: Intuit Inc.
- Inventors: Jiaxin ZHANG; Wendi CUI; Kamalika DAS; Sricharan KALLUR PALLI KUMAR
- Web page: https://patentgrep.com/patent/US20260064745A1

## CPC

- G06F 16/3347: {using vector based model}
- G06F 16/3326: {using relevance feedback from the user, e.g. relevance feedback on documents, documents sets, document terms or passages}
- G06F 16/3346: {using probabilistic model}
- G06F 16/383: using metadata automatically derived from the content
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

A method includes receiving a query from a user device and applying an embedding model to the query to generate a query vector data structure. A vector comparator is applied to the query vector data structure and embedded document chunk vector data structures to output a score measuring a semantic similarity between the vector data structures. The embedded document chunk vector data structures are generated by the embedding model being applied to documents in a knowledge base corpus. Responsive to the score failing to satisfy a threshold value, the query is rejected as being an out-of-distribution query by transmitting an electronic reject message. Responsive to the score satisfying the threshold value and indicating that the query is an in-distribution query, the language model is applied to the query and to the knowledge base corpus to output an answer. The method also includes returning the answer to the user device.

## Claims

### Claim 1

A method comprising:
receiving, at an embedding model executed by a processor, a user query from a user device;
applying the embedding model to the user query to generate a query vector data structure;
applying a vector comparator to the query vector data structure and a set of embedded document chunk vector data structures to output a similarity score, wherein:
the similarity score is a measure of a semantic similarity between the query vector data structure and the set of embedded document chunk vector data structures,
the set of embedded document chunk vector data structures are generated by the embedding model applied to a knowledge base corpus comprising a plurality of documents, and
each of the set of embedded document chunk vector data structures corresponds to an associated document of the plurality of documents;
determining that the similarity score fails to satisfy a threshold value;
identifying, responsive to the similarity score failing to satisfy the threshold value, the user query as being an out-of-distribution query;
transmitting an electronic reject message to the user device;
adding the user query to a query distribution data structure, the query distribution data structure representing a historic distribution of queries;
determining that a number of queries in the historic distribution of queries are outside a query threshold of the knowledge base corpus; and
updating, responsive to the number of queries being outside the query threshold, the knowledge base corpus by updating the plurality of documents in the knowledge base corpus.

### Claim 2 (depends on 1)

The method of claim 1, wherein the similarity score comprises a goodness-of-fit (GoF) test result between the query vector data structure and at least one embedded document chunk vector data structure.

### Claim 3 (depends on 1)

The method of claim 1, wherein the similarity score is the measure of the semantic similarity between the query vector data structure and an embedded document chunk vector data structure from the set of embedded document chunk vector data structures that is nearest in embedded vector space to the query vector data structure.

### Claim 4 (depends on 1)

The method of claim 1, wherein the similarity score is an average of measurements of the semantic similarity between the query vector data structure and a subset of embedded document chunk vector data structures from the set of embedded document chunk vector data structures.

### Claim 5 (depends on 1)

The method of claim 1, wherein applying the vector comparator to the query vector data structure and the set of embedded document chunk vector data structures comprises computing an energy score of the user query with respect to a subset of embedded document chunk vector data structures from the set of embedded document chunk vector data structures.

### Claim 6 (depends on 1)

The method of claim 1, further comprising adding an added document to the plurality of documents, wherein the added document is relevant to the user query.

### Claim 7

(canceled)

### Claim 8

(canceled)

### Claim 9

(canceled)

### Claim 10 (depends on 1)

The method of claim 1, further comprising transmitting an alert to the user device, responsive to the query threshold being outside the knowledge base corpus, that a shift in query distribution has occurred.

### Claim 11 (depends on 1)

The method of claim 1, further comprising:
applying a language model to the plurality of documents to generate synthetic in-knowledge queries; and
adding the synthetic in-knowledge queries to the query distribution data structure.

### Claim 12

A system comprising:
a server comprising a processor;
a data repository in communication with the processor, and storing:
a user query,
a query vector data structure, wherein the query vector data structure represents the user query in a vector space,
a knowledge base corpus, the knowledge base corpus comprising a plurality of documents,
a set of embedded document chunk vector data structures, wherein each of the set of embedded document chunk vector data structures represents an associated document of the plurality of documents,
a threshold value,
a query threshold value,
a query distribution data structure representing a historic distribution of queries,
an electronic reject message, and
an embedded model, wherein the processor is programmed to apply the embedded model to a document to output a vector data structure;
a vector comparator, wherein the processor is programmed to apply the vector comparator to a plurality of vectors to output a similarity score, the similarity score is a measure of a semantic similarity between the plurality of vectors;
a language model, wherein the processor is programmed to apply the language model to the vector data structure to output a natural language text string; and
a server controller executable by the processor to perform a computer-implemented method comprising:
receiving the user query from a user device;
applying the embedding model to the user query to generate the query vector data structure;
applying the vector comparator to the query vector data structure and the set of embedded document chunk vector data structures to output the similarity score;
determining that the similarity score fails to satisfy the threshold value;
identifying, responsive to the similarity score failing to satisfy the threshold value, the user query as being an out-of-distribution query;
transmitting an electronic reject message to the user device;
adding the user query to the query distribution data structure;
determining that a number of queries in the historic distribution of queries are outside a query threshold of the knowledge base corpus; and
updating, responsive to the number of queries being outside the query threshold, the knowledge base corpus by updating the plurality of documents in the knowledge base corpus.

### Claim 13 (depends on 12)

The system of claim 12, wherein the similarity score is the measure of the semantic similarity between the query vector data structure and an embedded document chunk vector data structure from the set of embedded document chunk vector data structures that is nearest in embedded vector space to the query vector data structure.

### Claim 14 (depends on 12)

The system of claim 12, wherein applying the vector comparator to the query vector data structure and the set of embedded document chunk vector data structures comprises computing an energy score of the user query with respect to a subset of embedded document chunk vector data structures from the set of embedded document chunk vector data structures.

### Claim 15 (depends on 12)

The system of claim 12, wherein the computer-implemented method further comprises:
adding an added document to the plurality of documents, wherein the added document is relevant to the user query. based on the similarity score.

### Claim 16

(canceled)

### Claim 17

(canceled)

### Claim 18 (depends on 12)

The system of claim 12, wherein the computer-implemented method further comprises:
transmitting an alert to the user device, responsive to the query threshold being outside the knowledge base corpus, that a shift in query distribution has occurred.

### Claim 19 (depends on 12)

The system of claim 12, further comprising:
applying the language model to the plurality of documents to generate synthetic in-knowledge queries; and
adding the synthetic in-knowledge queries to the query distribution data structure.

### Claim 20

(canceled)
