Information extraction from unstructured documents with hybrid retrieval augmentation using multi-modal language models

A system for extracting particular information from a document, the system comprising one or more processing circuits configured to:

• receive a payload that includes document text from the document, wherein the payload is generated from an optical character recognition tool;
• generate one or more chunks from the document text, a respective chunk of the one or more chunks associated with a corresponding portion of the document;
• store a mapping between the respective chunk and an image-based portion of the document comprising the document text from the respective chunk;
• identify a relevant chunk of the one or more chunks based on a search criterion related to a first prompt to extract the particular information;
• retrieve the corresponding portion of the document based on the mapping, wherein the corresponding portion of the document comprises an image-based representation of a page of the document;
• provide the first prompt or a second prompt based on the first prompt to a multi-modal language model, with the corresponding portion of the document retrieved based on the mapping and associated with the relevant chunk;
• and
• store a result from the multi-modal language model to the first prompt or the second prompt provided.

A system for extracting particular information from an image-based document, the system comprising:

• one or more processors;
• and
• one or more tangible, non-transitory memories configured to communicate with the one or more processors,
• the one or more tangible, non-transitory memories having instructions stored thereon that, in response to execution by the one or more processors, cause the one or more processors to:
• receive a payload that includes document text from the image-based document, wherein the payload is generated from an optical character recognition tool;
• generate one or more chunks from the document text;
• store, for each respective chunk of the one or more chunks, a mapping between the respective chunk and a corresponding page of the image-based document, the corresponding page comprising the document text of the respective chunk;
• identify a relevant chunk of the one or more chunks based on a search criterion related to a first prompt to extract the particular information;
• retrieve the corresponding page for the relevant chunk based on the mapping, wherein the corresponding portion of the document comprises an image-based representation of a page of the document;
• provide the first prompt or a second prompt based on the first prompt to a multi-modal language model, with the corresponding page of the image-based document retrieved based on the mapping and associated with the relevant chunk;
• and
• store a result from the multi-modal language model to the first prompt or the second prompt provided.