System and method for multimodal retrieval augmented generation for visually rich documents

A system for multimodal retrieval augmented generation for visual documents having spatial and visual features, the system comprising:

• a processor;
• and
• a memory operatively connected to the processor via a communication interface, the memory storing computer readable instructions, when executed, causes the processor to:
• receive, as input, a visual document having a plurality of texts, spatial, and visual features;
• implement a page-wise chunking algorithm to chunk the visual document into a plurality of page chunks;
• input the plurality of page chunks onto a trained embedding model;
• generate, by the embedding model, one vector for each page chunk in response to inputting the plurality of page chunks onto the embedding model, wherein each vector represents corresponding text, spatial, and visual feature of each page of the visual document;
• store all vectors corresponding to the plurality of the page chunks onto a database;
• input, in response to receiving a prompt of a query corresponding to the visual document, the vectors of the embedded chucks of pages retrieved from the database;
• identify most relevant chunks of pages included in the visual document based on the similarities between the query prompt and the chucks of pages;
• and
• generate, in response to inputting the most relevant chunks of pages onto a generative model, a response to the prompt corresponding to the visual document based on the identified most relevant chunks of pages.

A non-transitory computer readable medium configured to store instructions for multimodal retrieval augmented generation for visual documents having spatial and visual features, the instructions, when executed, cause a processor to perform the following:

• receiving, as input, a visual document having a plurality of texts, spatial and visual features;
• implementing a page-wise chunking algorithm to chunk the visual document into a plurality of page chunks;
• inputting the plurality of page chunks onto a trained embedding model;
• generating, by the embedding model, one vector for each page chunk in response to inputting the plurality of page chunks onto the embedding model, wherein each vector represents corresponding text, spatial, and visual feature of each page of the visual document;
• storing all vectors corresponding to the plurality of the page chunks onto a database;
• inputting, in response to receiving a prompt of a query corresponding to the visual document, the vectors of the embedded chucks of pages retrieved from the database;
• identifying most relevant chunks of pages included in the visual document based on the similarities between the query prompt and the chucks of pages;
• and
• generating, in response to inputting the most relevant chunks of pages onto a generative model, a response to the prompt corresponding to the visual document based on the identified most relevant chunks of pages.