Method and system for optimizing use of retrieval augmented generation pipelines in generative artificial intelligence applications

A system for adaptive context partitioning in a retrieval-augmented generation process, comprising:

• a processor;
• a communication device positioned in communication with the processor and configured to send and receive transmissions across a digital network;
• and
• a non-transitory computer-readable storage medium having stored thereon software that, when executed by the processor, is operable to:
• receive a query from a user;
• retrieve a plurality of relevant documents that form a combined context;
• monitor available system resources;
• dynamically adjust a partition size for context partitions based on the available system resources;
• partition the combined context into a plurality of context partitions according to the partition size;
• implement a map phase operable to generate a plurality of intermediate analysis results, wherein each context partition of the plurality of context partitions is processed in parallel;
• implement a reduce phase operable to synthesize the plurality of intermediate analysis results into a coherent response;
• and
• transmit the coherent response to the user.

A system for adaptive context partitioning in a retrieval-augmented generation system, comprising:

• means for receiving a query from a user;
• means for forming a combined context by retrieving a plurality of relevant documents from at least one document database based on the query;
• means for monitoring current system resources including processor availability and memory utilization;
• means for dynamically determining a partition size based on the current system resources;
• means for partitioning the combined context into a plurality of context partitions according to the partition size while preserving document boundaries to maintain semantic coherence;
• means for generating a plurality of intermediate analysis results by processing each context partition of the plurality of context partitions using a mapper prompt through one or more large language models (LLMs);
• means for generating a final response by processing the plurality of intermediate analysis results using a reducer prompt through the one or more LLMs;
• and
• means for transmitting the final response to the user.