System and method for general purpose pipeline for question answering on documents

A system for implementing a general-purpose question answering (QA) pipeline for question answering on documents across various domains, the system comprising:

• a processor;
• and
• a memory operatively connected to the processor via a communication interface, the memory storing computer readable instructions, when executed, causes the processor to:
• i) establish a communication link between a plurality of document sources having various domains and the general-purpose QA pipeline via communication interface, wherein the general-purpose QA pipeline includes an ingestion pipeline and an inference pipeline;
• ii) receive, via a receiving module included within the ingestion pipeline, a document among a set of plurality of documents received from the plurality of document sources;
• iii) chunk, via a chunking module included within the ingestion pipeline, the document into a plurality of chunks each including relatively smaller, semantically meaningful segment to address context length limitations;
• iv) embed, via an embedding module included within the ingestion pipeline, the plurality of chunks;
• v) store the embedded plurality of chunks into a vector database;
• repeat steps ii)-v) for all documents among the set of plurality of documents thereby creating a knowledge base;
• execute the inference pipeline that, given a query, retrieves, by a hybrid retriever module included within the inference pipeline, documents from the knowledge base that are relevant to the query by utilizing weighted combination of sparse and dense retrieval;
• implement a large language model (LLM)-based re-ranker module included within the inference pipeline to re-rank initial candidate chunks from the hybrid retriever module;
• implement an LLM-based answer generation module included within the inference pipeline to synthesize information relevant to answering question corresponding to the query from the re-ranked initial candidate chunks;
• and
• generate an answer corresponding to the query for consumption by downstream applications.

A non-transitory computer readable medium configured to store instructions for implementing a general-purpose question answering (QA) pipeline for question answering on documents across various domains, the instructions, when executed, cause a processor to perform the following:

• i) establishing a communication link between a plurality of document sources having various domains and the general-purpose QA pipeline via a communication network, wherein the general-purpose QA pipeline includes an ingestion pipeline and an inference pipeline;
• ii) receiving, by a receiving module included within the ingestion pipeline, a document among a set of plurality of documents received from the plurality of document sources;
• iii) chunking, by a chunking module included within the ingestion pipeline, the document into a plurality of chunks each including relatively smaller, semantically meaningful segment to address context length limitations;
• iv) embedding, by an embedding module included within the ingestion pipeline, the plurality of chunks;
• v) storing the embedded plurality of chunks into a vector database;
• repeating steps ii)-v) for all documents among the set of plurality of documents thereby creating a knowledge base;
• executing the inference pipeline that, given a query, retrieves, by a hybrid retriever module included within the inference pipeline, documents from the knowledge base that are relevant to the query by utilizing weighted combination of sparse and dense retrieval;
• implementing a large language model (LLM)-based re-ranker module included within the inference pipeline to re-rank initial candidate chunks from the hybrid retriever module;
• implementing an LLM-based answer generation module included within the inference pipeline to synthesize information relevant to answering question corresponding to the query from the re-ranked initial candidate chunks;
• and
• generating an answer corresponding to the query for consumption by downstream applications.