Document processing and retrieval for knowledge-based question answering

A system comprising:

• one or more processors;
• and
• one or more computer-readable media storing instructions which, when executed by the one or more processors, cause the system to perform the following operations:
• accessing a document, the document comprising text;
• dividing the document into a plurality of chunks of text;
• indexing the plurality of chunks, the indexing comprising storing each chunk mapped to respective identifying metadata, the identifying metadata including a chunk index for each chunk and a knowledge group of a plurality of knowledge groups for each chunk, each knowledge group indicating subject matter of a group of the plurality of chunks;
• receiving a query and an indication of a user-specified knowledge group via a graphical user interface (GUI);
• analyzing the query and the user-specified knowledge group to identify a subset of the chunks with subject matter similar to the query;
• comparing the subset of the chunks to the query to identify a chunk that is most similar to the query;
• formulating a prompt including the query, the identified chunk, and a subsequent chunk with a subsequent chunk index immediately following a chunk index of the identified chunk;
• providing the prompt to a language model, thereby causing the language model to generate a response to the query including information from the identified chunk;
• receiving, from the language model, the generated response;
• and
• returning, responsive to the received query, the generated response or a derivative thereof.

One or more non-transitory computer-readable media storing instructions which, when executed by one or more processors, cause a system to perform the following operations:

• accessing a document, the document comprising text;
• dividing the document into a plurality of chunks of text;
• indexing the plurality of chunks, the indexing comprising storing each chunk mapped to respective identifying metadata, the identifying metadata including a chunk index for each chunk and a knowledge group of a plurality of knowledge groups for each chunk, each knowledge group indicating subject matter of a group of the plurality of chunks;
• receiving a query and an indication of a user-specified knowledge group via a graphical user interface (GUI);
• analyzing the query and the user-specified knowledge group to identify a subset of the chunks with subject matter similar to the query;
• comparing the subset of the chunks to the query to identify a chunk that is most similar to the query;
• formulating a prompt including the query, the identified chunk, and a subsequent chunk with a subsequent chunk index immediately following a chunk index of the identified chunk;
• providing the prompt to a language model, thereby causing the language model to generate a response to the query including information from the identified chunk;
• receiving, from the language model, the generated response;
• and
• returning, responsive to the received query, the generated response or a derivative thereof.