# US 2026/0064698 A1 · Multimodal Data Ingestion And Retrieval For Agent Systems

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2026/0064698 A1
- Document ID: US20260064698A1
- Status: Published application
- Kind: A1
- Application number: 19/082,403
- Filed: 2025-03-18
- Published: 2026-03-05
- Earliest claimed: 2024-09-05
- Assignees: Oracle International Corporation
- Applicants: Oracle International Corporation
- Inventors: Xin Zhang; Zheng Wang; Yuying Wang; Genyi Huang; Mengqing Guo; Yazhe Hu; Zhonghai Deng; Yimo Liu; Rongguang Wang; Tao Sheng
- Web page: https://patentgrep.com/patent/US20260064698A1

## CPC

- G06F 16/3347: {using vector based model}
- G06F 16/24575: {using context}
- G06F 16/24578: {using ranking}
- G06F 16/3344: {using natural language analysis}
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

Techniques for multimodal document retrieval are disclosed herein. Multimodal documents that include both textual and graphical components are retrieved from a knowledge base by a multimodal retrieval augmented generation (RAG) agent in response to a query. The documents and/or components or chunks thereof are retrievable by the RAG agent from the knowledge base using the semantic summaries and/or vector search of embeddings in the knowledge base that are generated from text extracted from processing non-textual components of the data. The RAG agent classifies the query type to determine whether to use a semantic match for text or image summaries, full text semantic search, vector cosine similarity search, and/or other multimodal vector search. The RAG agent performs types of searches selected based on the modality used to generate the response to the query.

## Claims

### Claim 1

A method, comprising:
accessing a query;
identifying an embedding of at least a chunk of a multimodal document by performing a multimodal vector search based on the query;
generating a prompt including the chunk as context; and
generating a response to the query that is based on a result of inputting the prompt into a multimodal large language model;
wherein the method is performed by at least one device including a hardware processor.

### Claim 2 (depends on 1)

The method of claim 1, further comprising:
identifying a first document chunk of the multimodal document based on a semantic similarity between a text corpus of the first document chunk and the query;
identifying a second document chunk of the multimodal document based on a vector search using the query to identify a cross-modal embedding of an image or graphical data; and
generating a response to the query based on ranking the first document chunk and the second document chunk.

### Claim 3 (depends on 1)

The method of claim 1, further comprising:
accessing a second query;
inputting the second query into a classification model to determine a class of the second query based on an output of the classification model;
responsive to the class indicating text modality, identifying a second chunk of a second document using a semantic search;
responsive to the class indicating another modality, identifying the second chunk by performing a multimodal vector search based on the second query;
generating a second prompt including the second document as context; and
generating a second response to the second query that is based on a second result of inputting the second prompt into the multimodal large language model.

### Claim 4 (depends on 1)

The method of claim 1, further comprising:
accessing a second query;
identifying a modality keyword included in the second query;
responsive to the modality keyword indicating text modality, identifying a second chunk of a second document using a semantic search;
responsive to the modality keyword indicating another modality, identifying the second chunk by performing a multimodal vector search based on the second query;
generating a second prompt including the second document as context; and
generating a second response to the second query that is based on a result of inputting the second prompt into a multimodal large language model.

### Claim 5 (depends on 1)

The method of claim 1, further comprising:
matching an embedding of the query to an embedding of the multimodal document based on a cosine similarity score for the embedding of the query and the embedding of at least a chunk of the multimodal document.

### Claim 6 (depends on 3)

The method of claim 3, wherein:
the query includes an image component and a text component;
the method further comprising:
generating an embedding of the query by cross-embedding the image component and the text component into a vector representation; and
identifying an embedding of a chunk of a multimodal document by performing a multimodal vector search based on the vector representation.

### Claim 7 (depends on 1)

The method of claim 1, further comprising:
performing the multimodal vector search comprises identifying a multimodal cross-embedding of (a) a textual component of a document and (b) a non-textual component of the document; wherein
the textual component comprises a summary of the non-textual component.

### Claim 8 (depends on 7)

The method of claim 7, further comprising:
accessing a text summary of the non-textual component;
accessing an embedding of the non-textual component;
evaluating a semantic match between the text summary and the query to identify a first document chunk;
evaluating a cosine similarity score for the embedding of the non-textual component and the embedding of the query to identify a second document chunk;
weighting a first similarity score for the first document chunk using a first weight; and
weighting a second a similarity score for the second document chunk using a second weight.

### Claim 9 (depends on 1)

The method of claim 1, further comprising:
generating the response to the query by:
performing a semantic search of text based on the query to identify a first result;
performing a vector search of multimodal embeddings performed based on the query to identify a second result;
ranking the first result and the second result; and
inputting the query, the first result, the second result, the ranking, and the prompt into the multimodal large language model to obtain at least a portion of the response.

### Claim 10 (depends on 1)

The method of claim 1, wherein:
the multimodal document comprises a text component, an image component, and a graphical data component:
the embedding comprises a vector representation of the text component, the image component, the graphical data component, a first text description of the image component, and a second text description of the graphical data component.

### Claim 11 (depends on 10)

The method of claim 10, wherein
the first text description of the image component includes text identified using optical character recognition on the image component.

### Claim 12 (depends on 1)

The method of claim 1, further comprising:
responsive to determining that the query includes a query text and an image;
performing text component extraction on the image to generate a description of the image;
generating a cross-embedding of the description;
retrieving a document chunk by performing a search that is based on the cross-embedding; and
generating the response based on the document chunk.

### Claim 13 (depends on 1)

The method of claim 1, further comprising:
responsive to determining that the query includes a query text and graphical data;
performing text component extraction on the graphical data to generate a text description of the graphical data;
generating a cross-embedding of the text description;
retrieving a document chunk by performing a search that is based on the cross-embedding; and
generating the response based on the document chunk.

### Claim 14 (depends on 13)

The method of claim 13, further comprising:
responsive to determining that the query includes a query text, an image, and graphical data:
generating a cross-embedding of the query text, a description of the image, and a summary of the graphical data.

### Claim 15 (depends on 1)

The method of claim 1, further comprising:
accessing a second query;
obtaining a first search result using a semantic search based on the second query;
responsive to the first search result meeting a criteria:
performing a multimodal vector search based on the second query to identify a second chunk;
generating a second prompt including the second chunk as context; and
generating a second response to the second query that is based on a second result of inputting the second prompt into the multimodal large language model.

### Claim 16 (depends on 1)

The method of claim 1, further comprising:
inputting the query into a retrieval model to generate an identification of a data source based on the query;
retrieving the embedding from the data source; and
decoding the embedding to result in the chunk.

### Claim 17 (depends on 1)

The method of claim 1, further comprising;
generating the prompt based on the query and the chunk as context using a first language model;
accessing feedback for the first language model that is based on the response; and
fine-tuning the multimodal large language model or the first language model using the feedback as training data.

### Claim 18 (depends on 1)

The method of claim 1, wherein:
the multimodal vector search comprises a multi-head attention mechanism to weigh a first document chunk of a first modality and a second document chunk of a second modality.

### Claim 19

One or more non-transitory computer readable media comprising instructions which, when executed by one or more hardware processors, cause performance of operations comprising:
accessing a query;
identifying an embedding of at least a chunk of a multimodal document by performing a multimodal vector search based on the query to identify a multimodal cross-embedding of (a) a textual component of the multimodal document and (b) a non-textual component of the multimodal document
generating a prompt including the chunk as context; and
generating a response to the query that is based on a result of inputting the prompt into a multimodal large language model.

### Claim 20

A system, comprising:
at least one device including a hardware processor;
the system being configured to perform operations comprising:
accessing a query;
identifying an embedding of at least a chunk of a multimodal document by performing a multimodal vector search based on the query;
generating a prompt including the chunk as context; and
generating a response to the query that is based on a result of inputting the prompt into a multimodal large language model.
