Supervised text retrieval

A method for training a text retrieval system for embedding based retrieval of text, comprising:

• encoding text from a corpus into corpus text embeddings with an embedding model that is pretrained, wherein the embedding model converts text from user queries into baseline query embeddings;
• receiving a labeled retrieval dataset comprising training queries and corresponding training texts, wherein the training texts are added in the corpus;
• encoding the training queries into training query embeddings and the training texts into training text embeddings;
• storing the training query embeddings in a training query embedding index;
• storing the corpus text embeddings and the training text embeddings as text embeddings in a text embedding index;
• and
• training a query embedding transformation model using the labeled retrieval dataset to transform the baseline query embeddings produced by the embedding model into modified query embeddings based on a distance between the baseline query embeddings and the training query embeddings.

A method for embedding based retrieval of text with a text retrieval system, comprising:

• receiving a user query via an electronic interface;
• encoding the user query into a baseline query embedding with an embedding model, wherein the embedding model encodes text from a corpus into corpus text embeddings and the corpus text embeddings are stored as text embeddings in a text embedding index;
• transforming the baseline query embedding to a modified query embedding with a query embedding transformation model, the query embedding transformation model is trained based on a labeled retrieval dataset comprising training queries and corresponding training texts, wherein the training queries are converted by the embedding model to training query embeddings that are stored in a training query embedding index, and wherein the training texts are added to the corpus and are converted into training text embeddings by the embedding model and are stored as the text embeddings in the text embedding index, the query embedding transformation model transforms the baseline query embeddings into the modified query embeddings based on a distance between the baseline query embeddings and the training query embeddings;
• retrieving one or more texts from the corpus based on a nearest neighbor search of the text embeddings in the text embedding index using the modified query embedding;
• and
• providing the one or more texts and the user query to a prompt constructor for a Large Language Model (LLM) in a Retrieval Augmented Generation application that produces a prompt to the LLM that integrates the one or more texts and the user query.