# US 2026/0094465 A1 · Computationally efficient artifact tagging for document management

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2026/0094465 A1
- Document ID: US20260094465A1
- Status: Published application
- Kind: A1
- Application number: 19/287,726
- Filed: 2025-07-31
- Published: 2026-04-02
- Earliest claimed: 2024-09-27
- Assignees: Intuit Inc.
- Applicants: Intuit Inc.
- Inventors: Conrad Larson DE PEUTER; Harsha ILAPAKURTY; Jadiel DE ARMAS; Tharathorn RIMCHALA; Seth Harris RUBIN; Subathra SUNDARARAJAN
- Web page: https://patentgrep.com/patent/US20260094465A1

## CPC

- G06V 30/414: Extracting the geometrical structure, e.g. layout tree; Block segmentation, e.g. bounding boxes for graphics or text
- G06F 16/164: {File meta data generation}
- G06F 16/93: Document management systems
- G06T 3/4038: Image mosaicing, e.g. composing plane images from plane sub-images
- G06V 30/418: Document matching, e.g. of document images
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

Image pages are generated from a document. Assembling the image pages generates a collage. Two-dimensional text and bounding boxes are extracted from the image pages. A structure verbalizer spatially formats the two-dimensional text in one-dimension with spatial information to generate spatial-formatted text. The spatial-formatted text is concatenated to generate a text extraction. A multimodal embedding model is applied to the collage and the text extraction to generate a target artifact vector. The target artifact vector is compared against a set of preexisting artifact vectors to identify a corresponding artifact vector associated with a corresponding document having a corresponding metadata tag. A distance is determined between the corresponding artifact vector and the target artifact vector. Responsive to the distance being within a threshold distance additional steps are performed, including performing both applying the corresponding metadata tag to the document to generate a modified document and outputting the modified document.

## Claims

### Claim 1

A method comprising:
generating a plurality of image pages from a target document;
generating an image collage by assembling the plurality of image pages into a sequential image mosaic comprising a plurality of pixels;
extracting two-dimensional text and corresponding bounding boxes from the plurality of image pages;
applying a structure verbalizer to spatially format the two-dimensional text in one-dimension with spatial information based on the corresponding bounding boxes to generate spatial-formatted text;
concatenating the spatial-formatted text to generate a sequential spatial-formatted text extraction;
applying a multimodal embedding model to the plurality of pixels of the image collage and the sequential spatial-formatted text extraction to generate a target artifact vector comprising an embedded representation of the plurality of pixels and the sequential spatial-formatted text extraction;
comparing, using a nearest-neighbor model, the target artifact vector against each preexisting artifact vector in a set of preexisting artifact vectors to identify a corresponding artifact vector from the set of preexisting artifact vectors, wherein the corresponding artifact vector is associated with a corresponding document, the corresponding document having a corresponding metadata tag; and
determining that a multimodal embedding distance between the corresponding artifact vector and the target artifact vector are within a threshold distance and, in response to determining, performing:
applying the corresponding metadata tag to the target document to generate a modified target document, and
outputting the modified target document.

### Claim 2 (depends on 1)

The method of claim 1, wherein:
concatenating the spatial-formatted text to generate the sequential spatial-formatted text extraction comprises concatenating the spatial-formatted text from a first page,
the spatial-formatted text is from a subsequent page, and
a page break character separates the spatial-formatted text from the first page and separates the spatial-formatted text from the subsequent page.

### Claim 3 (depends on 1)

The method of claim 1, wherein the corresponding artifact vector has a similar embedded representation that is similar to the embedded representation of the plurality of pixels and the sequential spatial-formatted text extraction.

### Claim 4 (depends on 1)

The method of claim 1, further comprising:
confirming whether the corresponding metadata tag matches a portion of the two-dimensional text.

### Claim 5 (depends on 4)

The method of claim 4, further comprising:
indicating, in response to the corresponding metadata tag not matching the portion of the two-dimensional text, that the corresponding metadata tag comprises a hallucination of a language model.

### Claim 6 (depends on 1)

The method of claim 1, wherein the corresponding artifact vector is a preexisting artifact vector that is closest in a multimodal embedding space to the target artifact vector based on Euclidean distance.

### Claim 7 (depends on 1)

The method of claim 1, wherein the corresponding artifact vector is a preexisting artifact vector that is closest in a multimodal embedding space to the target artifact vector based on nearest neighbor voting.

### Claim 8 (depends on 1)

The method of claim 1, further comprising:
flagging, in response to the multimodal embedding distance between the corresponding artifact vector and the target artifact vector not being within the threshold distance, the target document as an anomalous document.

### Claim 9 (depends on 8)

The method of claim 8, further comprising taking additional steps in response to flagging the target document as the anomalous document, the additional steps comprising:
applying a large language model to the target document to generate a created metadata tag;
adding the created metadata tag to the target document to generate the modified target document; and
outputting the modified target document.

### Claim 10 (depends on 1)

The method of claim 1, further comprising:
applying optical character recognition to the plurality of image pages to extract the two-dimensional text.

### Claim 11 (depends on 1)

The method of claim 1, wherein the image collage comprises a grid of the plurality of image pages, the grid in sequence from left-to-right, top-to-bottom.

### Claim 12

A system comprising:
a server comprising a processor;
a data repository in communication with the processor, and storing:
a target document, the target document comprising two-dimensional text,
an image collage comprising a sequential image mosaic having a plurality of pixels,
spatial-formatted text comprising the two-dimensional text spatially formatted in one-dimension,
a sequential spatial-formatted text extraction,
a target artifact vector comprising an embedded representation of the plurality of pixels and the sequential spatial-formatted text extraction,
a set of preexisting artifact vectors, wherein each of set of preexisting artifact vectors has an embedded representation, and is associated with a corresponding document,
a threshold distance, and
a modified target document;
a structure verbalizer which, when executed by the processor, is programmed to apply the structure verbalizer to spatially format the two-dimensional text in one-dimension with spatial information based on corresponding bounding boxes from a plurality of image pages to generate the spatial-formatted text;
a multimodal embedding model which, when executed by the processor, is programmed to apply the multimodal embedding model to the plurality of pixels of the image collage and the sequential spatial-formatted text extraction to generate the target artifact vector; and
a server controller which, when executed by the processor, performs a computer-implemented method comprising:
generating the plurality of image pages from the target document,
generating the image collage by assembling the plurality of image pages into the sequential image mosaic,
extracting the two-dimensional text and corresponding bounding boxes from the image pages,
applying the structure verbalizer to the two-dimensional text to generate the spatial-formatted text,
concatenating the spatial-formatted text to generate the sequential spatial-formatted text extraction,
applying the multimodal embedding model to the plurality of pixels of the image collage and the sequential spatial-formatted text extraction to generate the target artifact vector,
comparing, using a nearest-neighbor model, the target artifact vector against each preexisting artifact vector in the set of preexisting artifact vectors to identify a corresponding artifact vector from the set of preexisting artifact vectors, wherein the corresponding artifact vector:
has a similar embedded representation that is similar to the embedded representation of the plurality of pixels and the sequential spatial-formatted text extraction, and
is associated with a corresponding document, the corresponding document having a corresponding metadata tag; and
determining that a multimodal embedding distance between the corresponding artifact vector and the target artifact vector is within the threshold distance and, in response to determining, performing additional steps comprising:
adding the corresponding metadata tag to the target document to generate the modified target document; and
outputting the modified target document.

### Claim 13 (depends on 12)

The system of claim 12, wherein the computer-implemented method further comprises:
concatenating the spatial-formatted text from a first page, the spatial-formatted text from a subsequent page and a page break character separating the spatial-formatted text from the first page and the spatial-formatted text from the subsequent page.

### Claim 14 (depends on 12)

The system of claim 12, wherein the computer-implemented method further comprises:
confirming whether the corresponding metadata tag matches a portion of the two-dimensional text.

### Claim 15 (depends on 14)

The system of claim 14, wherein the computer-implemented method further comprises:
indicating, in response to the corresponding metadata tag not matching the portion of the two-dimensional text, that the corresponding metadata tag comprises a hallucination of a language model.

### Claim 16 (depends on 12)

The system of claim 12, wherein the computer-implemented method further comprises:
flagging, in response to the multimodal embedding distance between the corresponding artifact vector and the target artifact vector not being within the threshold distance, the target document as an anomalous document; and
performing, in response to flagging the target document as the anomalous document, additional steps comprising:
applying a large language model to the target document to generate a created metadata tag;
adding the created metadata tag to the target document to generate the modified target document; and
outputting the modified target document.

### Claim 17 (depends on 12)

The system of claim 12, wherein the computer-implemented method further comprises:
applying optical character recognition to the plurality of image pages to extract the two-dimensional text.

### Claim 18 (depends on 12)

The system of claim 12, wherein the image collage comprises a grid of the plurality of image pages.

### Claim 19 (depends on 18)

The system of claim 18, wherein the grid is in sequence from left-to-right, top-to-bottom.

### Claim 20

A method comprising:
generating a plurality of image pages from a target document;
generating an image collage by assembling the plurality of image pages into a sequential image mosaic comprising a plurality of pixels;
extracting two-dimensional text and corresponding bounding boxes from the plurality of image pages;
applying a structure verbalizer to spatially format the two-dimensional text in one-dimension with spatial information based on the corresponding bounding boxes to generate spatial-formatted text;
concatenating the spatial-formatted text to generate a sequential spatial-formatted text extraction;
applying a multimodal embedding model to the plurality of pixels of the image collage and the sequential spatial-formatted text extraction to generate a target artifact vector comprising an embedded representation of the plurality of pixels and the sequential spatial-formatted text extraction;
comparing using a nearest-neighbor model to the target artifact vector against each preexisting artifact vector in a set of preexisting artifact vectors to identify a corresponding artifact vector from the set of preexisting artifact vectors, wherein the corresponding artifact vector:
has a similar embedded representation that is similar to the embedded representation of the plurality of pixels and the sequential spatial-formatted text extraction, and
is associated with a corresponding document, the corresponding document having a corresponding metadata tag; and
performing, in response to a multimodal embedding distance between the corresponding artifact vector and the target artifact vector not being within a threshold distance, additional steps comprising:
flagging the target document as an anomalous document,
applying a large language model to the target document to generate a created metadata tag,
adding the created metadata tag to the target document to generate a modified target document, and
outputting the modified target document.
