Data listing search utilizing large language models

A system comprising:

• a memory;
• and
• a processing device operatively coupled to the memory, the processing device to:
• receive a search query comprising a set of search terms;
• retrieve a plurality of data listings based on the search terms of the search query, merging the plurality of data listings to eliminate duplicates;
• compare a first embedding generated by a large language model (LLM) from the search query to second embeddings generated by the LLM for each of the plurality of data listings, the second embeddings filtered according to a nearest-neighbors search to eliminate multiple instances of similar data listings of the plurality of data listings based at least on a weighting strategy, wherein the weighting strategy combines a scoring of elements of the similar data listings of the plurality of data listings to rank the plurality of data listings, wherein a top rank listing is maintained and the similar data listings remaining include relevance information that are listed as less relevant in view of the top rank listing, such that the similar data listings remaining are listed as less relevant or are eliminated, and wherein the second embeddings include database schema information associated with the data listings, the database schema information comprising at least one of a schema name, a table name, or a column name, to determine a respective relevance for each of the plurality of data listings to the search query;
• and
• rank the plurality of data listings based on the respective relevance for each of the plurality of data listings to the search query.

A non-transitory computer-readable medium having instructions stored thereon which, when executed by a processing device, cause the processing device to:

• receive a search query comprising a set of search terms;
• retrieve a plurality of data listings based on the search terms of the search query, merging the plurality of data listings to eliminate duplicates;
• compare, by the processing device, a first embedding generated by a large language model (LLM) from the search query to second embeddings generated by the LLM for each of the plurality of data listings, the second embeddings filtered according to a nearest-neighbors search to eliminate multiple instances of similar data listings of the plurality of data listings based at least on a weighting strategy, wherein the weighting strategy combines a scoring of elements of the similar data listings of the plurality of data listings to rank the plurality of data listings, wherein a top rank listing is maintained and the similar data listings remaining include relevance information that are listed as less relevant in view of the top rank listing, such that the similar data listings remaining are listed as less relevant by the processing device or are eliminated, and wherein the second embeddings include database schema information associated with the data listings, the database schema information comprising at least one of a schema name, a table name, or a column name, to determine a respective relevance for each of the plurality of data listings to the search query;
• and
• rank the plurality of data listings based on the respective relevance for each of the plurality of data listings to the search query.