# US 2026/0228229 A1 · Computing systems and methods for ranking a set of passages based on their relevance to a query using an llm

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2026/0228229 A1
- Document ID: US20260228229A1
- Status: Published application
- Kind: A1
- Application number: 19/043,200
- Filed: 2025-01-31
- Published: 2026-08-06
- Earliest claimed: 2025-01-31
- Applicants: The Toronto-Dominion Bank
- Inventors: Noël VOUITSIS; Jiapeng WU; Zhaoyan LIU; Yi SUI; Shiyi HOU; Graham Andrew WARNER; Maksims VOLKOVS
- Web page: https://patentgrep.com/patent/US20260228229A1

## CPC

- G06F 16/24578: {using ranking}
- G06F 16/3344: {using natural language analysis}
- G06F 16/3334: {Selection or weighting of terms from queries, including natural language queries}
- G06F 16/3347: {using vector based model}
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

Methods and systems for ranking a set of passages based on their relevance to a query. The methods comprise providing a set of one or more ranking prompt to a first LLM which causes the first LLM to generate a ranking of a test set of passages with respect to a test query; using a second LLM to generate an evaluation metric based on the ranking of the test set of passages relative to a reference ranking; using a set of one or more LLMs to generate, based on the evaluation metric, textual feedback on the set of one or more ranking prompt; generating a modified set of one or more ranking prompt based on the textual feedback; and providing the modified set of one or more ranking prompt to the first LLM which causes the first LLM to rank the set of passages with respect to the query.

## Claims

### Claim 1

A system for ranking a set of passages based on their relevance to a query, the system comprising:
a memory, a communication interface, and at least one processor operatively coupled to the memory and the communication interface;
the at least one processor configured to:
(a) provide a set of one or more ranking prompt to a first large language model (LLM) which causes the first LLM to generate a ranking of a test set of passages based on their relevance to a test query;
(b) execute a second LLM to generate an evaluation metric indicating a quality of the ranking of the test set of passages relative to a reference ranking of the test set of passages;
(c) use a set of one or more LLMs to generate, based on the evaluation metric, textual feedback on the set of one or more ranking prompt;
(d) generate a modified set of one or more ranking prompt based on the textual feedback; and
(e) provide the modified set of one or more ranking prompt to the first LLM which causes the first LLM to rank the set of passages based on their relevance to the query.

### Claim 2 (depends on 1)

The system of claim 1, wherein using the set of one or more LLMs to generate, based on the evaluation metric, the textual feedback on the set of one or more ranking prompt comprises backpropagating the evaluation metric to the set of one or more ranking prompt using the set of one or more LLMs to generate a textual gradient of the evaluation metric with respect to the set of one or more ranking prompt.

### Claim 3 (depends on 1)

The system of claim 1, wherein using the set of one or more LLMs to generate, based on the evaluation metric, the textual feedback on the set of one or more ranking prompt comprises (i) using the set of one or more LLMs to generate textual feedback on the ranking of the test set of passages based on the evaluation metric, and (ii) using the set of one or more LLMs to generate the textual feedback on the set of one or more ranking prompt based on the textual feedback on the ranking of the test set of passages.

### Claim 4 (depends on 3)

The system of claim 3, wherein using the set of one or more LLMs to generate the textual feedback on the ranking of the test set of passages based on the evaluation metric comprises providing an LLM of the set of one or more LLMs, the evaluation metric, inputs to the second LLM which caused the second LLM to generate the evaluation metric, and a first feedback prompt that causes the LLM of the set of one or more LLMs to generate the textual feedback on the ranking of the test set of passages.

### Claim 5 (depends on 4)

The system of claim 4, wherein the inputs to the second LLM which caused the second LLM to generate the evaluation metric comprises the ranking of the test set of passages, the reference ranking of the test set of passages, and an evaluation prompt which caused the second LLM to generate the evaluation metric.

### Claim 6 (depends on 4)

The system of claim 4, wherein using the set of one or more LLMs to generate the textual feedback on the set of one or more ranking prompt based on the textual feedback on the ranking of the test set of passages comprises providing an LLM of the set of one or more LLMs with the textual feedback on the ranking of the test set of passages, the ranking of the test set of passages, inputs to the first LLM which caused the first LLM to generate the ranking of the test set of passages, and a second feedback prompt that causes the LLM of the set of one or more LLMs to generate the textual feedback on the set of one or more ranking prompt.

### Claim 7 (depends on 6)

The system of claim 6, wherein the inputs to the first LLM which caused the first LLM to generate the ranking of the test set of passages comprises the test set of passages, the test query, and the set of one or more ranking prompt.

### Claim 8 (depends on 1)

The system of claim 1, wherein generating the modified set of one or more ranking prompt based on the textual feedback comprises using a third LLM to generate the modified set of one or more ranking prompt based on the set of one or more ranking prompt and the textual feedback.

### Claim 9 (depends on 8)

The system of claim 8, wherein using the third LLM to generate the modified set of one or more ranking prompt based on the set of one or more ranking prompt and the textual feedback comprising providing the third LLM the set of one or more ranking prompt, the textual feedback and an optimizer prompt which instructs the third LLM to generate the modified set of one or more ranking prompt based on the one or more ranking prompt and the textual feedback.

### Claim 10 (depends on 1)

The system of claim 1, wherein each passage in the test set of passages and the set of passages comprises all or a portion of a document in a corpus of documents.

### Claim 11 (depends on 10)

The system of claim 10, wherein the at least one processor is configured to execute (a) to (c) for each of a plurality of different test set of passages from the corpus of documents to generate textual feedback on the set of one or more ranking prompt for each of the plurality of different test set of passages; and the modified set of one or more ranking prompt is based on the textual feedback on the set of one or more ranking prompt for each of the plurality of different test set of passages.

### Claim 12 (depends on 1)

The system of claim 1, wherein the at least one processor is configured to, prior to executing (e), repeat (a) to (d) with the modified set of one or more ranking prompt.

### Claim 13 (depends on 1)

The system of claim 1, wherein the textual feedback on the set of one or more ranking prompt comprises interpretable natural language describing how the set of one or more ranking prompt should be changed to improve the evaluation metric.

### Claim 14 (depends on 1)

The system of claim 1, wherein the set of one or more ranking prompt comprises a system prompt and modifying the set of one or more ranking prompt comprises modifying the system prompt.

### Claim 15 (depends on 1)

The system of claim 1, wherein the at least one processor is further configured to:
receive the query; and
generate a response to the query based on the ranking of the set of passages based on their relevance to the query.

### Claim 16 (depends on 15)

The system of claim 15, wherein the at least one processor is further configured to:
use an information retrieval system to identify, from a corpus of documents, the set of passages, wherein the passages in the set of passages are relevant to the query; and
wherein generating the response to the query based on the ranking of the set of passages based on their relevance to the query comprises:
selecting a subset of the set of passages based on the ranking of the set of passages; and
using an LLM to generate the response to the query based on the subset of the set of passages.

### Claim 17

A method for ranking a set of passages based on their relevance to a query, the method executed in a computing environment comprising at least one processor, a communication interface, and memory, and the method comprising:
(a) providing a set of one or more ranking prompt to a first large language model (LLM) which causes the first LLM to generate a ranking of a test set of passages based on their relevance to a test query;
(b) executing a second LLM to generate an evaluation metric indicating a quality of the ranking of the test set of passages relative to a reference ranking of the test set of passages;
(c) using a set of one or more LLMs to generate, based on the evaluation metric, textual feedback on the set of one or more ranking prompt;
(d) generating a modified set of one or more ranking prompt based on the textual feedback; and
(e) providing the modified set of one or more ranking prompt to the first LLM which causes the first LLM to rank the set of passages based on their relevance to the query.

### Claim 18 (depends on 17)

The method of claim 17, wherein using the set of one or more LLMs to generate, based on the evaluation metric, the textual feedback on the set of one or more ranking prompt comprises (i) using the set of one or more LLMs to generate textual feedback on the ranking of the test set of passages based on the evaluation metric, and (ii) using the set of one or more LLMs to generate the textual feedback on the set of one or more ranking prompt based on the textual feedback on the ranking of the test set of passages.

### Claim 19 (depends on 17)

The method of claim 17, wherein generating the modified set of one or more ranking prompt based on the textual feedback comprises using a third LLM to generate the modified set of one or more ranking prompt based on the set of one or more ranking prompt and the textual feedback.

### Claim 20

A non-transitory computer readable medium storing computer executable instructions which, when executed by at least one computer processor, cause the at least one computer processor to carry out a method for ranking a set of passages based on their relevance to a query, the method comprising:
(a) providing a set of one or more ranking prompt to a first large language model (LLM) which causes the first LLM to generate a ranking of a test set of passages based on their relevance to a test query;
(b) executing a second LLM to generate an evaluation metric indicating a quality of the ranking of the test set of passages relative to a reference ranking of the test set of passages;
(c) using a set of one or more LLMs to generate, based on the evaluation metric, textual feedback on the set of one or more ranking prompt;
(d) generating a modified set of one or more ranking prompt based on the textual feedback; and
(e) providing the modified set of one or more ranking prompt to the first LLM which causes the first LLM to rank the set of passages based on their relevance to the query.
