# US 2024/0202530 A1 · Systems and methods for unsupervised training in text retrieval tasks

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2024/0202530 A1
- Document ID: US20240202530A1
- Status: Published application
- Kind: A1
- Application number: 18/303,313
- Filed: 2023-04-19
- Published: 2024-06-20
- Earliest claimed: 2022-12-15
- Assignees: Salesforce, Inc.
- Applicants: Salesforce Inc.
- Inventors: Rui Meng; Yingbo Zhou; Ye Liu; Semih Yavuz; Ning Yu
- Web page: https://patentgrep.com/patent/US20240202530A1

## CPC

- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06F 40/20: Natural language analysis (semantic analysis of natural language G06F40/30)
- G06F 40/40: Processing or translation of natural language (natural language analysis G06F40/20; semantic analysis G06F40/30)
- G06N 3/0455: Auto-encoder networks; Encoder-decoder networks
- G06N 3/088: Non-supervised learning, e.g. competitive learning
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

Embodiments described herein provide systems and methods for training a text retrieval model. A system may generate queries associated with provided documents. The queries may be generated in one or more different manners. Examples of query generation may include extracting relevant spans of text from the documents, prompting a language model for a topic, title, abstractive summary, and/or extractive summary based on the documents. Metadata such as title or other HTML tags may be used as queries. Using the one or more queries, the text retrieval model may be trained using contrastive learning, using the generated query, and positive and negative sample documents. A fine-tuning training phase may be performed using domain-specific data which may also be done with generated query pairs, or may be done in a supervised fashion with provided queries. The text retrieval model may be used to locate documents given an input query.

## Claims

### Claim 1

A method of training a text retrieval model, the method comprising:
receiving, via a data interface, a plurality of text documents;
generating, by a processor, a query corresponding to at least one text document from the plurality of text documents, wherein the generating includes at least one of:
(a) extracting a text span from the at least one text document as the query based on a relevance level between the extracted text span and the at least one text document; or
(b) generating, by a pre-trained language model, a text output as the query based on an input of the at least one text document conditioned with a pre-defined prompt;
selecting a negative sample document from the plurality of text documents;
computing a loss objective based on the query, the at least one text document, and the negative sample document; and
training the text retrieval model by updating parameters of the text retrieval model based on the computed loss objective via backpropagation.

### Claim 2 (depends on 1)

The method of claim 1, wherein the plurality of text documents is a first plurality of text documents, and the loss objective is a first loss objective, further comprising:
receiving, via the data interface, a second plurality of text documents;
computing a second loss objective using at least one text document of the second plurality of text documents as a finetuning dataset; and
finetuning the trained text retrieval model by updating the parameters of the text retrieval model based on the computed second loss objective.

### Claim 3 (depends on 2)

The method of claim 2, wherein the second plurality of text documents are annotated with a plurality of queries, respectively, and,
wherein the second loss objective is a contrastive loss based on the plurality of queries and the second plurality of text documents.

### Claim 4 (depends on 2)

The method of claim 2, further comprising:
generating, by the processor, a second query corresponding to at least one text document from the second plurality of text documents,
wherein the computing the second loss objective is further based on the second query.

### Claim 5 (depends on 1)

The method of claim 1, further comprising:
generating multiple queries corresponding to the at least one text document, wherein the multiple queries are generated in more than one manner.

### Claim 6 (depends on 1)

The method of claim 1, wherein the text span is further extracted from a metadata from the at least one text document.

### Claim 7 (depends on 1)

The method of claim 1, wherein the pre-defined prompt includes a prompt associated with at least one of:
a topic,
a title,
an extractive summary, or
an abstractive summary.

### Claim 8 (depends on 1)

The method of claim 1, further comprising:
training the text retrieval model using queries generated in different manners at different training iterations.

### Claim 9 (depends on 8)

The method of claim 8, wherein the training is performed iteratively, wherein a relative frequency of using queries generated in different manners is determined based on a preselected value.

### Claim 10

A system for training a text retrieval model, the system comprising:
a memory that stores the text retrieval model and a plurality of processor executable instructions;
a communication interface that receives a plurality of text documents; and
one or more hardware processors that read and execute the plurality of processor-executable instructions from the memory to perform operations comprising:
generating, by a processor, a query corresponding to at least one text document from the plurality of text documents, wherein the generating includes at least one of:
(a) extracting a text span from the at least one text document as the query based on a relevance level between the extracted text span and the at least one text document; or
(b) generating, by a pre-trained language model, a text output as the query based on an input of the at least one text document conditioned with a pre-defined prompt;
selecting a negative sample document from the plurality of text documents;
computing a loss objective based on the query, the at least one text document, and the negative sample document; and
training the text retrieval model by updating parameters of the text retrieval model based on the computed loss objective via backpropagation.

### Claim 11 (depends on 10)

The system of claim 10, wherein the plurality of text documents is a first plurality of text documents, and the loss objective is a first loss objective, the operations further comprising:
receiving, via a data interface, a second plurality of text documents;
computing a second loss objective using at least one text document of the second plurality of text documents as a finetuning dataset; and
finetuning the trained text retrieval model by updating the parameters of the text retrieval model based on the computed second loss objective.

### Claim 12 (depends on 11)

The system of claim 11, wherein the second plurality of text documents are annotated with a plurality of queries, respectively, and,
wherein the second loss objective is a contrastive loss based on the plurality of queries and the second plurality of text documents.

### Claim 13 (depends on 11)

The system of claim 11, the operations further comprising:
generating, by the processor, a second query corresponding to at least one text document from the second plurality of text documents,
wherein the computing the second loss objective is further based on the second query.

### Claim 14 (depends on 10)

The system of claim 10, the operations further comprising:
generating multiple queries corresponding to the at least one text document, wherein the multiple queries are generated in more than one manner.

### Claim 15 (depends on 10)

The system of claim 10, wherein the text span is further extracted from a metadata from the at least one text document.

### Claim 16 (depends on 10)

The system of claim 10, wherein the pre-defined prompt includes a prompt associated with at least one of:
a topic,
a title,
an extractive summary, or
an abstractive summary.

### Claim 17 (depends on 10)

The system of claim 10, the operations further comprising:
training the text retrieval model using queries generated in different manners at different training iterations.

### Claim 18 (depends on 17)

The system of claim 17, wherein the training is performed iteratively, wherein a relative frequency of using queries generated in different manners is determined based on a preselected value.

### Claim 19

A non-transitory machine-readable medium comprising a plurality of machine-executable instructions which, when executed by one or more processors, are adapted to cause the one or more processors to perform operations comprising:
receiving, via a data interface, a plurality of text documents;
generating, by a processor, a query corresponding to at least one text document from the plurality of text documents, wherein the generating includes at least one of:
(a) extracting a text span from the at least one text document as the query based on a relevance level between the extracted text span and the at least one text document; or
(b) generating, by a pre-trained language model, a text output as the query based on an input of the at least one text document conditioned with a pre-defined prompt;
selecting a negative sample document from the plurality of text documents;
computing a loss objective based on the query, the at least one text document, and the negative sample document; and
training a text retrieval model by updating parameters of the text retrieval model based on the computed loss objective via backpropagation.

### Claim 20 (depends on 19)

The non-transitory machine-readable medium of claim 19, wherein the plurality of text documents is a first plurality of text documents, and the loss objective is a first loss objective, further comprising:
receiving, via the data interface, a second plurality of text documents;
computing a second loss objective using at least one text document of the second plurality of text documents as a finetuning dataset; and
finetuning the trained text retrieval model by updating the parameters of the text retrieval model based on the computed second loss objective.
