# US 2026/0099698 A1 · System and method for multimodal retrieval augmented generation for visually rich documents

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2026/0099698 A1
- Document ID: US20260099698A1
- Status: Published application
- Kind: A1
- Application number: 18/906,898
- Filed: 2024-10-04
- Published: 2026-04-09
- Earliest claimed: 2024-10-04
- Assignees: JPMorgan Chase Bank, N.A.
- Applicants: JPMorgan Chase Bank, N.A.
- Inventors: Simerjot KAUR; Zhiqiang MA; Mathieu SIBUE; Farima FARMAHINIFARAHANI; Lawrence YONG; Dongsheng WANG; Armineh NOURBAKHSH; Lucas CECCHI
- Web page: https://patentgrep.com/patent/US20260099698A1

## CPC

- G06N 3/0455: Auto-encoder networks; Encoder-decoder networks
- G06F 16/93: Document management systems
- G06F 40/289: Phrasal analysis, e.g. finite state techniques or chunking
- G06F 40/30: Semantic analysis
- G06N 3/045: Combinations of networks
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

Various methods and processes, apparatuses/systems, and media for multimodal retrieval augmented generation for visually rich documents are disclosed. A processor implements a page-wise chunking algorithm to chunk a visual document into a plurality of page chunks; inputs the plurality of page chunks onto a trained embedding model; generates one vector for each page chunk. Each vector represents corresponding text, spatial and visual feature of each page of the visual document; inputs, in response to receiving a prompt of a query corresponding to the visual document, the vectors of the embedded chucks of pages retrieved from the database; identifies most relevant chunks of pages based on the similarities between the query prompt and chucks of pages; and generates, in response to inputting the most relevant chunks of pages onto a generative model, a response to the prompt corresponding to the visual document based on the identified most relevant chunks of pages.

## Claims

### Claim 1

A method for multimodal retrieval augmented generation for visual documents having spatial and visual features by utilizing one or more processors along with allocated memory, the method comprising:
receiving, as input, a visual document having a plurality of texts, spatial, and visual features;
implementing a page-wise chunking algorithm to chunk the visual document into a plurality of page chunks;
inputting the plurality of page chunks onto a trained embedding model;
generating, by the embedding model, one vector for each page chunk in response to inputting the plurality of page chunks onto the embedding model, wherein the vector represents corresponding texts, spatial, and visual feature of each page of the visual document;
storing all vectors corresponding to the plurality of the page chunks onto a database;
inputting, in response to receiving a prompt of a query corresponding to the visual document, the vectors of the embedded chucks of pages retrieved from the database;
identifying most relevant chunks of pages included in the visual document based on the similarities between the query prompt and the chucks of pages; and
generating, in response to inputting the most relevant chunks of pages onto a generative model, a response to the prompt corresponding to the visual document based on the identified most relevant chunks of pages.

### Claim 2 (depends on 1)

The method according to claim 1, wherein the generative model is a trained prompt generation model that utilizes a pre-trained autoregressive large language model, and wherein in generating the response to the prompt, the method further comprising:
using the trained prompt-generation model to generate query space for each page chunk from the plurality page chunks;
capturing a semantic essence of the prompt by implementing an alignment learning algorithm utilizing an autoencoder; and
outputting a response to the prompt corresponding to the captured semantic essence given the context of most relevant chunks of pages.

### Claim 3 (depends on 2)

The method according to claim 2, the method further comprising:
fine-tuning the prompt-generation model by training the prompt-generation model that maximizes log-likelihood of correctly predicting a next token in the sequence of tokens from the prompt, given a context of document page tokens.

### Claim 4 (depends on 2)

The method according to claim 2, in capturing the semantic essence of the prompt, the method further comprising:
fine-tuning an autoencoder to encode prompt by using contrastive learning objective; and
obtaining the prompt semantics from a first pooling token output from the autoencoder corresponding to the prompt.

### Claim 5 (depends on 1)

The method according to claim 1, further comprising:
identifying most relevant chunks included in the visual document based on the vectors by applying a cosine similarity algorithm.

### Claim 6 (depends on 5)

The method according to claim 5, wherein a distance between two vectors measures their relatedness in a manner such that small distances correspond to high relatedness and large distances correspond to low relatedness.

### Claim 7 (depends on 5)

The method according to claim 5, the method further comprising:
ranking most relevant chunks of pages that correspond to high relatedness.

### Claim 8

A system for multimodal retrieval augmented generation for visual documents having spatial and visual features, the system comprising:
a processor; and
a memory operatively connected to the processor via a communication interface, the memory storing computer readable instructions, when executed, causes the processor to:
receive, as input, a visual document having a plurality of texts, spatial, and visual features;
implement a page-wise chunking algorithm to chunk the visual document into a plurality of page chunks;
input the plurality of page chunks onto a trained embedding model;
generate, by the embedding model, one vector for each page chunk in response to inputting the plurality of page chunks onto the embedding model, wherein each vector represents corresponding text, spatial, and visual feature of each page of the visual document;
store all vectors corresponding to the plurality of the page chunks onto a database;
input, in response to receiving a prompt of a query corresponding to the visual document, the vectors of the embedded chucks of pages retrieved from the database;
identify most relevant chunks of pages included in the visual document based on the similarities between the query prompt and the chucks of pages; and
generate, in response to inputting the most relevant chunks of pages onto a generative model, a response to the prompt corresponding to the visual document based on the identified most relevant chunks of pages.

### Claim 9 (depends on 8)

The system according to claim 8, wherein the generative model is a trained prompt generation model that utilizes a pre-trained autoregressive large language model, and wherein, in generating the response to the prompt, the processor is further configured to:
use the trained prompt-generation model to generate query space for each page chunk from the plurality page chunks;
capture a semantic essence of the prompt by implementing an alignment learning algorithm utilizing an autoencoder; and
output a response to the prompt corresponding to the captured semantic essence given the context of most relevant chunks of pages.

### Claim 10 (depends on 9)

The system according to claim 9, wherein the processor is further configured to:
fine-tune the prompt-generation model by training the prompt-generation model that maximizes log-likelihood of correctly predicting a next token in the sequence of tokens from the prompt, given a context of document page tokens.

### Claim 11 (depends on 9)

The system according to claim 9, wherein, in capturing the semantic essence of the prompt, the processor is further configured to:
fine-tune an autoencoder to encode prompt by using contrastive learning objective; and
obtain the prompt semantics from a first pooling token output from the autoencoder corresponding to the prompt.

### Claim 12 (depends on 9)

The system according to claim 9, the processor is further configured to:
identify most relevant chunks included in the visual document based on the vectors by applying a cosine similarity algorithm.

### Claim 13 (depends on 12)

The system according to claim 12, wherein a distance between two vectors measures their relatedness in a manner such that small distances correspond to high relatedness and large distances correspond to low relatedness.

### Claim 14 (depends on 13)

The system according to claim 13, the processor is further configured to:
rank most relevant chunks of pages that correspond to high relatedness.

### Claim 15

A non-transitory computer readable medium configured to store instructions for multimodal retrieval augmented generation for visual documents having spatial and visual features, the instructions, when executed, cause a processor to perform the following:
receiving, as input, a visual document having a plurality of texts, spatial and visual features;
implementing a page-wise chunking algorithm to chunk the visual document into a plurality of page chunks;
inputting the plurality of page chunks onto a trained embedding model;
generating, by the embedding model, one vector for each page chunk in response to inputting the plurality of page chunks onto the embedding model, wherein each vector represents corresponding text, spatial, and visual feature of each page of the visual document;
storing all vectors corresponding to the plurality of the page chunks onto a database;
inputting, in response to receiving a prompt of a query corresponding to the visual document, the vectors of the embedded chucks of pages retrieved from the database;
identifying most relevant chunks of pages included in the visual document based on the similarities between the query prompt and the chucks of pages; and
generating, in response to inputting the most relevant chunks of pages onto a generative model, a response to the prompt corresponding to the visual document based on the identified most relevant chunks of pages.

### Claim 16 (depends on 15)

The non-transitory computer readable medium according to claim 15, wherein the generative model is a trained prompt-generation model that utilizes a pre-trained autoregressive large language model, and wherein, in generating the response to the prompt, the instructions, when executed cause the processor to further perform the following:
using the trained prompt-generation model to generate vectors of query space for each page chunk from the plurality page chunks;
capturing a semantic essence of the prompt by implementing an alignment learning algorithm utilizing an autoencoder; and
outputting a response to the prompt corresponding to the captured semantic essence given the context of most relevant chunks of pages.

### Claim 17 (depends on 16)

The non-transitory computer readable medium according to claim 16, wherein the instructions, when executed cause the processor to further perform the following:
fine-tuning the prompt-generation model by training the prompt-generation model that maximizes log-likelihood of correctly predicting a next token in the sequence of tokens from the prompt, given a context of document page tokens.

### Claim 18 (depends on 16)

The non-transitory computer readable medium according to claim 16, wherein, in capturing the semantic essence of the prompt, the instructions, when executed cause the processor to further perform the following:
fine-tuning an autoencoder to encode prompt by using contrastive learning objective; and obtaining the prompt semantics from a first pooling token output from the autoencoder corresponding to the prompt.

### Claim 19 (depends on 15)

The non-transitory computer readable medium according to claim 15, wherein the instructions, when executed cause the processor to further perform the following:
identifying most relevant chunks included in the visual document based on the vectors by applying a cosine similarity algorithm, wherein a distance between two vectors measures their relatedness in a manner such that small distances correspond to high relatedness and large distances correspond to low relatedness.

### Claim 20 (depends on 19)

The non-transitory computer readable medium according to claim 19, wherein the instructions, when executed cause the processor to further perform the following:
ranking most relevant chunks of pages that correspond to high relatedness.
