Ranking-augmented generation for long documents

A system comprising:

• (a) at least one hardware processor;
• and
• (b) a non-transitory computer-readable storage medium having program code embodied therewith, the program code executable by said at least one hardware processor to:
• receive, as input, a user query and a source document intended for a content-grounded question-answering or multi-turn conversation task by a specified large language model (LLM) which has a context window size limit, wherein said source document has a size which exceeds said context window size limit;
• divide said source document into a plurality of segments;
• generate a prompt instructing a second language model, different from the specified LLM, to generate a per-segment response for each of said plurality of segments, wherein said prompt:
• (i) frames a task in which said second language model generates a conversation between a user and an agent based on the content of the respective segment, and
• (ii) specifies that said per-segment response is to be generated based on the content of the respective segment;
• apply said second language model to each of said segments in accordance with said prompt, to generate a respective per-segment response for each of said segments;
• assign, by said second language model, to each of said segments a relevance score based on a computed similarity between said per-segment response generated for said segment and said user query;
• select the k-top segments having the highest said relevance scores;
• combine said selected k-top segments into a virtual document having a size which complies with said context window size limit;
• and
• feed said virtual document as input to the specified LLM, to generate a response that is grounded in the content of said virtual document.

A computer program product comprising a non-transitory computer-readable storage medium having program code embodied therewith, the program code executable by at least one hardware processor to:

• receive, as input, a user query and a source document intended for a content-grounded question-answering or multi-turn conversation task by a specified large language model (LLM) which has a context window size limit, wherein said source document has a size which exceeds said context window size limit;
• divide said source document into a plurality of segments;
• generate a prompt instructing a second language model, different from the specified LLM, to generate a per-segment response for each of said plurality of segments, wherein said prompt:
• (i) frames a task in which said second language model generates a conversation between a user and an agent based on the content of the respective segment, and
• (ii) specifies that said per-segment response is to be generated based on the content of the respective segment;
• apply said second language model to each of said segments in accordance with said prompt, to generate a respective per-segment response for each of said segments;
• assign, by said second language model, to each of said segments a relevance score based on a computed similarity between said per-segment response generated for said segment and said user query;
• select the k-top segments having the highest said relevance scores;
• combine said selected k-top segments into a virtual document having a size which complies with said context window size limit;
• and
• feed said virtual document as input to the specified LLM, to generate a response that is grounded in the content of said virtual document.