# US 2026/0030480 A1 · Effective Multi-Modal Retrieval-Augmented Generation (RAG) Agent With Twin-Database And Comprehensive Multi-Format Data Ingestion

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2026/0030480 A1
- Document ID: US20260030480A1
- Status: Published application
- Kind: A1
- Application number: 19/001,309
- Filed: 2024-12-24
- Published: 2026-01-29
- Earliest claimed: 2024-07-29
- Assignees: Oracle International Corporation
- Applicants: Oracle International Corporation
- Inventors: Liyu Gong; Yuying Wang
- Web page: https://patentgrep.com/patent/US20260030480A1

## CPC

- G06N 3/045: Combinations of networks
- G06N 3/09: Supervised learning
- G06F 16/90: Details of database functions independent of the retrieved data types
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

Techniques for ingesting and using content items by a Retrieval-Augmented Generation (RAG) agent are disclosed. A RAG agent accesses content items that include textual data and/or non-textual image data (e.g., a table, a chart, a document, or a picture). When the RAG agent detects that content items include non-textual image data, the RAG agent invokes a large multimodal model (LMM) that is configured to classify the non-textual image data into a variety of classifications. The RAG agent also classifies the non-textual image data. Using this classification as selection criteria, the RAG agent selects an LMM that corresponds to the classification from a set of available LMMs. The RAG agent ensures that the selected LMM is configured to generate text from non-textual image data that corresponds to the classification. The generated text and extracted image data are both used by the RAG agent to respond to queries.

## Claims

### Claim 1

One or more non-transitory computer readable media comprising instructions which, when executed by one or more hardware processors, cause performance of operations comprising:
accessing a plurality of content items, wherein the plurality of content items comprises:
one or more content items comprising non-textual image data; and
one or more content items comprising textual data;
in response to detecting that the plurality of content items includes non-textual image data, invoking a classification LMM configured to classify non-textual image data into one of a plurality of classifications; and
in response to the classification LMM detecting a first content item of the plurality of content items that comprises non-textual image data corresponding to a first classification of the plurality of classifications, selecting a first LMM from a plurality of LMMs, wherein the first LMM is configured to generate text from non-textual image data corresponding to the first classification.

### Claim 2 (depends on 1)

The non-transitory media of claim 1, wherein the non-textual image data includes one or more of:
a table;
a chart;
a document; or
a picture.

### Claim 3 (depends on 1)

The non-transitory media of claim 1, wherein:
the first content item comprises a table;
the first classification is associated with tables;
the first LMM is a table LMM;
the operations further comprise:
using the first LMM to generate table textual data from the first content item; and
using the table LMM to store table component data comprising information about table-specific components from the first content item.

### Claim 4 (depends on 2)

The non-transitory media of claim 2, wherein the operations further comprise:
in response to the classification LMM detecting a second content item of the plurality of content items that comprises non-textual image data corresponding to a chart:
selecting a chart LMM from the plurality of LMMs, wherein the chart LMM is configured to generate text from non-textual image data corresponding to charts; and
using the chart LMM to generate chart textual data from the second content item.

### Claim 5 (depends on 2)

The non-transitory media of claim 2, wherein the operations further comprise:
in response to the classification LMM detecting a third content item of the plurality of content items that comprises non-textual image data corresponding to a document:
selecting a document LMM from the plurality of LMMs, wherein the document LMM is configured to generate text from non-textual image data corresponding to documents; and
using the document LMM to generate document textual data from the third content item.

### Claim 6 (depends on 2)

The non-transitory media of claim 2, wherein the operations further comprise:
in response to the classification LMM detecting a fourth content item of the plurality of content items that comprises non-textual image data corresponding to a picture:
selecting a picture LMM from the plurality of LMMs, wherein the picture LMM is configured to generate text from non-textual image data corresponding to pictures; and
using the picture LMM to generate image textual data from the fourth content item.

### Claim 7 (depends on 2)

The non-transitory media of claim 2, wherein the first content item comprises both textual data and non-textual data, wherein the instructions further comprise:
extracting first textual data and second textual data from the first content item, wherein the first textual data occurs in the first content item before the table and the second textual data occurs after the table;
generating a table identifier corresponding to the table; and
storing a copy of the table using the table identifier as an index value.

### Claim 8 (depends on 7)

The non-transitory media of claim 7, wherein the instructions further comprise:
generating a first text string, wherein the first text string comprises the first textual data, the table textual data, the table identifier, and the second textual data.

### Claim 9 (depends on 8)

The non-transitory media of claim 8, wherein the first textual data is before the table textual data and the table identifier in the first text string, and the second textual data is after the table textual data and the table identifier in the first text string.

### Claim 10 (depends on 9)

The non-transitory media of claim 9, wherein the instructions further comprise:
chunking the first text string into corresponding chunks based at least in part on the anticipated size of the corresponding chunks; and
storing the chunks corresponding to the first text string in a text database.

### Claim 11 (depends on 10)

The non-transitory media of claim 10, wherein the instructions further comprise:
in response to a first query to a RAG agent comprising a RAG LMM, generating a first response based at least in part on the first text string, wherein the first response comprises the table and at least a portion of the table textual data, wherein generating the first response comprises fetching the table using the table identifier.

### Claim 12 (depends on 1)

The non-transitory media of claim 1, wherein:
the first content item comprises both textual data and non-textual data;
the first content item comprises a chart;
the first classification is associated with charts;
the first LMM is a chart LMM;
the operations further comprise:
using the first LMM to generate chart textual data from the first content item;
using the chart LMM to store table component data comprising information about chart-specific components from the first content item;
wherein the instructions further comprise:
extracting first textual data and second textual data from the first content item, wherein the first textual data occurs before the chart and the second textual data occurs after the chart;
generating a chart identifier corresponding to the chart;
storing a copy of the chart using the chart identifier as an index value; and
generating a first text string, wherein the second text string comprises the first textual data, the chart textual data, the chart identifier, and the second textual data, wherein the first textual data is before the chart textual data and the chart identifier in the first text string, and the second textual data is after the chart textual data and the chart identifier in the first text string.

### Claim 13 (depends on 12)

The non-transitory media of claim 12, wherein the instructions further comprise:
chunking the first text string into corresponding chunks based at least in part on the anticipated size of the corresponding chunks; and
storing the chunks corresponding to the first text string in a text database.

### Claim 14 (depends on 13)

The non-transitory media of claim 13, wherein the instructions further comprise:
in response to a first query to a RAG agent comprising a RAG LMM, generating a first response based at least in part on the first text string, wherein the first response comprises the chart and at least a portion of the chart textual data, wherein generating the first response comprises fetching the chart using the chart identifier.

### Claim 15 (depends on 1)

The non-transitory media of claim 1, wherein:
the first content item comprises both textual data and non-textual data;
the first content item comprises a document;
the first classification is associated with documents;
the first LMM is a document LMM;
the operations further comprise:
using the first LMM to generate document textual data from the first content item;
using the document LMM to store table component data comprising information about document-specific components from the first content item;
wherein the instructions further comprise:
extracting first textual data and second textual data from the first content item, wherein the first textual data occurs before the document and the second textual data occurs after the document;
generating a document identifier corresponding to the document;
storing a copy of the document using the document identifier as an index value; and
generating a first text string, wherein the second text string comprises the first textual data, the document textual data, the document identifier, and the second textual data, wherein the first textual data is before the document textual data and the document identifier in the first text string, and the second textual data is after the document textual data and the document identifier in the first text string.

### Claim 16 (depends on 13)

The non-transitory media of claim 13, wherein the instructions further comprise:
in response to a first query to a RAG agent comprising a RAG LMM, generating a first response based at least in part on the first text string, wherein the first response comprises the document and at least a portion of the document textual data, wherein generating the first response comprises fetching the document using the document identifier.

### Claim 17 (depends on 1)

The non-transitory media of claim 1, wherein:
the first content item comprises both textual data and non-textual data;
the first content item comprises a picture;
the first classification is associated with pictures;
the first LMM is a picture LMM;
the operations further comprise:
using the first LMM to generate picture textual data from the first content item;
using the picture LMM to store table component data comprising information about picture-specific components from the first content item;
wherein the instructions further comprise:
extracting first textual data and second textual data from the first content item, wherein the first textual data occurs before the picture and the second textual data occurs after the picture;
generating a picture identifier corresponding to the picture;
storing a copy of the picture using the picture identifier as an index value; and
generating a first text string, wherein the second text string comprises the first textual data, the picture textual data, the picture identifier, and the second textual data, wherein the first textual data is before the picture textual data and the picture identifier in the first text string, and the second textual data is after the picture textual data and the picture identifier in the first text string.

### Claim 18 (depends on 13)

The non-transitory media of claim 13, wherein the instructions further comprise:
in response to a first query to a RAG agent comprising a RAG LMM, generating a first response based at least in part on the first text string, wherein the first response comprises the picture and at least a portion of the picture textual data, wherein generating the first response comprises fetching the picture using the picture identifier.

### Claim 19

A method comprising:
accessing a plurality of content items, wherein the plurality of content items comprises:
one or more content items comprising non-textual image data; and
one or more content items comprising textual data;
in response to detecting that the plurality of content items includes non-textual image data, invoking a classification LMM configured to classify non-textual image data into one of a plurality of classifications; and
in response to the classification LMM detecting a first content item of the plurality of content items that comprises non-textual image data corresponding to a first classification of the plurality of classifications, selecting a first LMM from a plurality of LMMs, wherein the first LMM is configured to generate text from non-textual image data corresponding to the first classification;
wherein the method is performed by at least one device including a hardware processor.

### Claim 20

A system comprising:
at least one device including a hardware processor;
the system being configured to perform operations comprising:
accessing a plurality of content items, wherein the plurality of content items comprises:
one or more content items comprising non-textual image data; and
one or more content items comprising textual data;
in response to detecting that the plurality of content items includes non-textual image data, invoking a classification LMM configured to classify non-textual image data into one of a plurality of classifications; and
in response to the classification LMM detecting a first content item of the plurality of content items that comprises non-textual image data corresponding to a first classification of the plurality of classifications, selecting a first LMM from a plurality of LMMs, wherein the first LMM is configured to generate text from non-textual image data corresponding to the first classification.
