Preserving tabular data integrity for query processing system

A system comprising:

• a memory;
• and
• at least one processor coupled to the memory and configured to perform operations comprising:
• receiving a query to be executed against a knowledgebase;
• identifying one or more keywords from the query;
• performing a vector search against the knowledgebase based on the one or more keywords, wherein the knowledgebase comprises a plurality of documents, and wherein a first document of the plurality of documents has been divided into a plurality of chunks and stored in the knowledgebase;
• identifying a subset of chunks from the plurality of chunks related to generating an answer for the query based on the vector search, wherein the subset of chunks includes a first chunk, and wherein the first chunk comprises a table ID corresponding to a table;
• identifying a table image corresponding to the table ID, wherein the table image was included in the first document prior to being divided into the plurality of chunks, and wherein the table image is stored separately from the first chunk;
• generating a prompt for a language model to generate the answer to the query based on the subset of chunks, including the first chunk and the table image;
• and
• providing the answer, generated by the language model based on the subset of chunks, responsive to the receiving the query.

A non-transitory computer-readable medium having instructions stored thereon that, when executed by at least one computing device, cause the at least one computing device to perform operations comprising:

• receiving a query to be executed against a knowledgebase;
• identifying one or more keywords from the query;
• performing a vector search against the knowledgebase based on the one or more keywords, wherein the knowledgebase comprises a plurality of documents, and wherein a first document of the plurality of documents has been divided into a plurality of chunks and stored in the knowledgebase;
• identifying a subset of chunks from the plurality of chunks related to generating an answer for the query based on the vector search, wherein the subset of chunks includes a first chunk, and wherein the first chunk comprises a table ID corresponding to a table;
• identifying a table image corresponding to the table ID, wherein the table image was included in the first document prior to being divided into the plurality of chunks, and wherein the table image is stored separately from the first chunk;
• generating a prompt for a language model to generate the answer to the query based on the subset of chunks, including the first chunk and the table image;
• and
• providing the answer, generated by the language model based on the subset of chunks, responsive to the receiving the query.