# US 2026/0311503 A1 · System and method for automated document annotation

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2026/0311503 A1
- Document ID: US20260311503A1
- Status: Published application
- Kind: A1
- Application number: 19/170,312
- Filed: 2025-04-04
- Published: 2026-10-08
- Earliest claimed: 2025-04-04
- Assignees: Quantiphi, Inc
- Applicants: Quantiphi, Inc
- Inventors: Dagnachew Birru; Harikrishnan P M; Siddartha Reddy Nareddy; Venkata Sai Prakash Mukkamala; Saisubramaniam Gopalakrishnan; Goutham Vignesh Saravanan; Vishal Vaddina
- Web page: https://patentgrep.com/patent/US20260311503A1

## CPC

- G06V 30/19173: {Classification techniques}
- G06V 10/761: {Proximity, similarity or dissimilarity measures}
- G06V 10/764: using classification, e.g. of video objects
- G06V 10/768: {using context analysis, e.g. recognition aided by known co-occurring patterns}
- G06V 10/96: Management of image or video recognition tasks
- G06V 30/147: {Determination of region of interest}
- G06V 30/158: {using character size, text spacings or pitch estimation}
- G06V 30/19093: {Proximity measures, i.e. similarity or distance measures}
- G06V 30/262: using context analysis, e.g. lexical, syntactic or semantic context
- G06V 30/414: Extracting the geometrical structure, e.g. layout tree; Block segmentation, e.g. bounding boxes for graphics or text
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

Disclosed is a system for automated document annotation. The system comprises a processor. Moreover, the system comprises a memory storing instructions that, when executed by the processor, cause the processor to receive one or more documents and a set of labels to be annotated within the documents. Furthermore, the system processes the documents using a Large Vision Language Model (LVLM) to extract label-value information from the documents. Additionally, the system processes the documents using Optical Character Recognition (OCR) to extract text elements and corresponding bounding box information from the documents. Finally, the system performs Spatial Informed Contextual Matching (SICM) that utilizes both the extracted label-value information and the bounding box information to assign labels to the text elements based on textual similarity and spatial relationships between the text elements.

## Claims

### Claim 1

A system for automated document annotation, comprising:
a processor; and
a memory storing instructions that, when executed by the processor, cause the processor to:
receive one or more documents and a set of labels to be annotated within the documents;
process the documents using a Large Vision Language Model (LVLM) to extract label-value information from the documents;
process the documents using Optical Character Recognition (OCR) to extract text elements and corresponding bounding box information from the documents; and
perform Spatial Informed Contextual Matching (SICM) that utilizes both the extracted label-value information and the bounding box information to assign labels to the text elements based on textual similarity and spatial relationships between the text elements.

### Claim 2 (depends on 1)

The system of claim 1, wherein performing the SICM comprises:
determining textual similarity between the extracted label-value information and the extracted text elements;
identifying multiple candidate text elements having similar textual content; and
selecting an optimal text element from the multiple candidate text elements based on spatial relationships between the candidate text elements.

### Claim 3 (depends on 2)

The system of claim 2, wherein the spatial relationships include at least one of:
proximity between text elements, relative position of text elements, and layout structure of the document.

### Claim 4 (depends on 2)

The system of claim 2, wherein the processor is further caused to apply a text matching technique to determine the textual similarity, wherein the text matching technique comprises at least one of:
a fuzzy matching calculation and an edit distance calculation.

### Claim 5 (depends on 1)

The system of claim 1, wherein the processor is further caused to:
generate annotations for the documents, wherein each annotation associates a text element with a label and a corresponding bounding box.

### Claim 6 (depends on 1)

The system of claim 1, wherein the processor is further caused to:
receive a query regarding information contained in the documents;
extract an answer to the query using the LVLM;
identify a bounding box associated with the answer using the SICM; and
highlight the identified bounding box on the documents to provide visual reference identification for the answer.

### Claim 7 (depends on 1)

The system of claim 1, wherein the processor is further caused to:
generate a synthetic dataset of documents based on the received documents; and
automatically annotate the synthetic dataset using the LVLM, OCR, and SICM.

### Claim 8 (depends on 7)

The system of claim 7, wherein the processor is further caused to:
train a machine learning model using the annotated synthetic dataset to perform key information extraction (KIE) tasks.

### Claim 9 (depends on 1)

The system of claim 1, wherein the documents comprise at least one of:
digital text files, scanned images, forms, and invoices.

### Claim 10 (depends on 1)

The system of claim 1, wherein the LVLM is configured to process both image and text data simultaneously to extract the label-value information.

### Claim 11

A method for automated document annotation, comprising:
receiving one or more documents and a set of labels to be annotated within the documents;
extracting label-value information from the documents using a Large Vision Language Model (LVLM);
extracting text elements and corresponding bounding box information from the documents using Optical Character Recognition (OCR); and
performing Spatial Informed Contextual Matching (SICM) that utilizes both the extracted label-value information and the bounding box information to assign labels to the text elements based on textual similarity and spatial relationships between the text elements.

### Claim 12 (depends on 11)

The method of claim 11, wherein performing the SICM comprises:
determining textual similarity between the extracted label-value information and the extracted text elements;
for each label-value pair:
identifying multiple candidate text elements having similar textual content;
analysing spatial relationships between the candidate text elements; and
selecting an optimal text element from the multiple candidate text elements based on the spatial relationships.

### Claim 13 (depends on 12)

The method of claim 12, wherein analysing spatial relationships comprises computing at least one of:
distances between text elements, angles between text elements, and relative positions of text elements within the document layout.

### Claim 14 (depends on 11)

The method of claim 11, further comprising:
generating annotations for the documents, wherein each annotation associates a text element with a label and includes a corresponding bounding box; and
outputting the annotated documents.

### Claim 15 (depends on 11)

The method of claim 11, further comprising:
receiving a query regarding information contained in the documents;
extracting an answer to the query using the LVLM;
identifying a bounding box associated with the answer using the SICM; and
highlighting the identified bounding box on the documents to provide visual reference identification for the answer.

### Claim 16 (depends on 11)

The method of claim 11, further comprising:
generating a synthetic dataset of documents based on the received documents using at least one of:
templating, data augmentation, and generative models; and
automatically annotating the synthetic dataset using the LVLM, OCR, and SICM.

### Claim 17 (depends on 16)

The method of claim 16, further comprising:
training a specialized key information extraction (KIE) model using the annotated synthetic dataset; and
deploying the KIE model to perform information extraction tasks on new documents without requiring the LVLM.

### Claim 18 (depends on 11)

The method of claim 11, wherein extracting the label-value information using the LVLM comprises:
processing the documents as images to understand the document layout;
identifying visual cues related to the requested labels; and
extracting label-value pairs in a structured format.

### Claim 19 (depends on 11)

The method of claim 11, wherein determining textual similarity comprises applying at least one text matching technique selected from:
fuzzy matching, N-gram matching, and edit distance calculation.

### Claim 20 (depends on 11)

The method of claim 11, wherein the SICM assigns confidence scores to potential label-text element matches based on a combination of textual similarity and spatial relationship factors and selects matches with the highest confidence scores.
