# US 2026/0064949 A1 · Augmented language model for artifact extraction

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2026/0064949 A1
- Document ID: US20260064949A1
- Status: Published application
- Kind: A1
- Application number: 18/820,053
- Filed: 2024-08-29
- Published: 2026-03-05
- Earliest claimed: 2024-08-29
- Assignees: Intuit Inc.
- Applicants: Intuit Inc.
- Inventors: Richard Joel BECKER; Karelia Del Carmen PENA-PENA; Raj Brij SRIVASTAVA; Joey-Michael FALLONE
- Web page: https://patentgrep.com/patent/US20260064949A1

## CPC

- G06F 40/174: Form filling; Merging
- G06F 40/194: Calculation of difference between files
- G06V 10/811: {the classifiers operating on different input data, e.g. multi-modal recognition}
- G06V 30/412: Layout analysis of documents structured with printed lines or input boxes, e.g. business forms or tables
- G06V 30/413: Classification of content, e.g. text, photographs or tables
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

A method including applying an image processing application to an electronic document to generate image processing data describing artifacts in the electronic document. The method also includes applying a server controller to the image processing data to generate at least one relationship among the artifacts. The method also includes converting the at least one relationship and the artifacts into an object notation language data structure. The method also includes generating a multimodal prompt for a language model by combining the object notation language data structure, a reference to the electronic document, and a prompt template. The prompt template includes instructions for the language model to extract segments from the electronic document. The method also includes applying, according to the multimodal prompt, the language model to the electronic document to extract, from the electronic document, the segments and a segment relationship among the segments. The method also includes returning the segments.

## Claims

### Claim 1

A method comprising:
applying an image processing application to an electronic document to generate image processing data describing a plurality of artifacts in the electronic document;
applying a server controller to the image processing data to generate at least one relationship among the plurality of artifacts;
converting the at least one relationship and the plurality of artifacts into an object notation language data structure;
generating a multimodal prompt for a language model by combining the object notation language data structure, a reference to the electronic document, and a prompt template, wherein the prompt template comprises instructions for the language model to extract a plurality of segments from the electronic document;
applying, according to the multimodal prompt, the language model to the electronic document to extract, from the electronic document, the plurality of segments and a segment relationship among the plurality of segments; and
returning the plurality of segments.

### Claim 2 (depends on 1)

The method of claim 1, wherein returning the plurality of segments comprises:
storing the plurality of segments in a non-transitory computer readable storage medium,
building a knowledge graph using the plurality of segments, and
storing the knowledge graph in the non-transitory computer readable storage medium.

### Claim 3 (depends on 1)

The method of claim 1, wherein the electronic document comprises a form and wherein returning the plurality of segments comprises:
generating, using the plurality of segments and the segment relationship, a computer executable program for filling out the form, and
programming computer executable instructions for filling out the form according to the computer executable program.

### Claim 4 (depends on 1)

The method of claim 1, wherein the plurality of artifacts comprise at least one of a bounding box, a text string within a bounding box, a text string outside a bounding box, a bar code, a Quick Response (QR) code, a contour of the electronic document, a field, a checkbox, and an amount of pixel fill within a bounding box.

### Claim 5 (depends on 1)

The method of claim 1,
wherein the electronic document comprises a plurality of pixels, and
wherein applying the image processing application comprises analyzing the plurality of pixels to identify the plurality of artifacts.

### Claim 6 (depends on 5)

The method of claim 5, wherein analyzing the plurality of pixels comprises identifying contiguous pixels that form text, and further identifying a font size, font type, and a language formed by the text.

### Claim 7 (depends on 5)

The method of claim 5, wherein analyzing the plurality of pixels comprises identifying contiguous pixels that form a bounding box and a relationship of textual content relative to the bounding box.

### Claim 8 (depends on 5)

The method of claim 5,
wherein the at least one relationship and the plurality of artifacts comprise a bounding box and text associated with the bounding box, and
wherein converting comprises:
establishing a coordinate system for the electronic document,
identifying, in the object notation language data structure, the text as content,
reproducing, in the object notation language data structure, the text as being directly associated with the content,
identifying, in the object notation language data structure, the bounding box as a bound,
specifying, in the object notation language data structure, coordinates of the bounding box relative to the coordinate system, and
specifying, in the object notation language data structure, a relationship between the bounding box and the text.

### Claim 9 (depends on 1)

The method of claim 1, wherein generating the multimodal prompt comprises:
retrieving a prompt template comprising an instruction to analyze a specified electronic document,
adding, to the prompt template, the object notation language data structure, and
specifying that the specified electronic document is the electronic document.

### Claim 10 (depends on 9)

The method of claim 9, wherein generating the multimodal prompt further comprises:
generating additional instructions instructing the language model to format the plurality of segments and the segment relationship in a specified format that is specific to the at least one relationship and the plurality of artifacts, and
adding the additional instructions to the prompt template.

### Claim 11 (depends on 9)

The method of claim 9, wherein the prompt template further comprises a system message limiting how the language model is to analyze the specified electronic document.

### Claim 12 (depends on 9)

The method of claim 9, wherein the prompt template further comprises a system message limiting how the language model is to output a result of analyzing the specified electronic document.

### Claim 13 (depends on 1)

The method of claim 1, wherein applying the language model comprises:
executing, by a computer processor, the language model on the multimodal prompt and the electronic document.

### Claim 14

A method comprising
applying a server controller to determine a difference between a first dataset and a second dataset,
wherein the first dataset comprises a plurality of artifacts extracted from an electronic document by an image processing application, and
wherein the second dataset comprises a plurality of segments extracted from the electronic document by a language model;
augmenting, to generate an updated segment using the difference, at least one of the plurality of segments extracted by the language model, wherein augmenting at least comprises modifying the at least one of the plurality of segments using the first dataset; and
returning a modified plurality of segments comprising the plurality of segments in which the updated segment replaces the at least one of the plurality of segments.

### Claim 15 (depends on 14)

The method of claim 14, further comprising:
identifying a plurality of relationships among the plurality of segments; and
generating a graph data structure from the plurality of segments by specifying the plurality of segments as nodes and specifying the plurality of relationships as edges between the nodes.

### Claim 16 (depends on 14)

The method of claim 14, wherein augmenting further comprises adding metadata describing the plurality of segments.

### Claim 17 (depends on 14)

The method of claim 14, further comprising:
applying a graph database generator to the modified plurality of segments to generate a knowledge graph.

### Claim 18

A system comprising
a computer processor;
a data repository in communication with the computer processor and storing:
an electronic document,
a reference to the electronic document,
image processing data describing a plurality of artifacts in the electronic document,
at least one relationship among the plurality of artifacts,
an object notation language data structure,
a prompt template comprising natural language instructions to extract a plurality of segments from the electronic document,
a multimodal prompt, and
a segment relationship among the plurality of segments;
an image processing application programmed, when executed by the computer processor, to generate the image processing data from the electronic document;
a server controller programmed, when executed by the computer processor, to:
generate, from the image processing data, the at least one relationship among the plurality of artifacts,
convert the at least one relationship and the plurality of artifacts into the object notation language data structure,
generate the multimodal prompt by combining object notation language data structure, the reference to the electronic document, and the prompt template, and
return the plurality of segments; and
a language model programmed, when executed by the computer processor according to the multimodal prompt, to extract, from the electronic document, the plurality of segments and the segment relationship.

### Claim 19 (depends on 18)

The system of claim 18, further comprising:
a knowledge graph generator which, when executed by the computer processor, is programmed to generate a graph database from the plurality of segments and the segment relationship extracted from the electronic document.

### Claim 20 (depends on 18)

The system of claim 18,
wherein the electronic document comprises a form, and
wherein the system further comprises a program generator which, when executed by the computer processor, is programmed to generate, using the plurality of segments and the segment relationship, a computer executable program for filling out the form.
