# US 12,602,547 B2 · Domain adapting graph networks for visually rich documents > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,602,547 B2 - Document ID: US12602547B2 - Status: Granted - Kind: B2 - Application number: 18/240,480 - Filed: 2023-08-31 - Published: 2024-08-29 - Granted: 2026-04-14 - Earliest claimed: 2023-08-31 - Assignees: Oracle International Corporation - Applicants: Oracle International Corporation - Inventors: Amit Agarwal; Srikant Panda; Deepak Karmakar; Kulbhushan Pachauri - Web page: https://patentgrep.com/patent/US12602547B2 ## CPC - G06F 40/284: Lexical analysis, e.g. tokenisation or collocates - G06F 16/35: Clustering; Classification - G06N 3/045: Combinations of networks - G06N 3/08: Learning methods - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS ## Abstract In some implementations, techniques described herein may include identifying text in a visually rich document and determining a sequence for the identified text. The techniques may include selecting a language model based at least in part on the identified text and the determined sequence. Moreover, the techniques may include assigning each word of the identified text to a respective token to generate textual features corresponding to the identified text. The techniques may include extracting visual features corresponding to the identified text. The techniques may include determining positional features for each word of the identified text. The techniques may include generating a graph representing the visually rich document, each node in the graph representing each of the visual features, textual features, and positional features of a respective word of the identified text. The techniques may include training a classifier on the graph to classify each respective word of the identified text. ## Claims ### Claim 1 A computer-implemented method comprising: identifying, by a computing system, text in a visually rich document; determining, by the computing system, a sequence for the identified text, the sequence comprising a numerical order for each word of the identified text; selecting, by the computing system, a pretrained language model based at least in part on the identified text, the determined sequence, and a domain of the identified text; assigning, by the computing system, each word of the identified text to a respective token using the pretrained language model and the determined sequence to generate textual features corresponding to the identified text, each respective token comprising a string of one or more words; extracting, by the computing system, visual features corresponding to the identified text, the visual features comprising information about a plurality of pixels representing each word of the identified text; determining, by the computing system, positional features for each word of the identified text, the positional features comprising respective coordinates within the visually rich document for each word of the identified text, the respective coordinates for a word being coordinates corresponding to a position of the word or coordinates corresponding to a region of the visually rich document that contains the word; fusing, by the computing system, the textual features and the visual features to generate fused features; generating, by the computing system, a document model representing the visually rich document by assigning the positional features and the fused features to nodes in the document model, each node in the document model representing the fused features and the positional features of a respective word of the identified text; training, by the computing system, a classifier to classify each respective word of the identified text, the classifier being trained on the document model representing the visually rich document; and classifying, by the computing system, the respective word of the identified text with the classifier. ### Claim 2 (depends on 1) The method of claim 1, wherein the respective word is classified as a key or a value of a key value pair. ### Claim 3 (depends on 1) The method of claim 1, wherein the document model is a graph neural network. ### Claim 4 (depends on 1) The method of claim 1, wherein the domain comprises a language or a subject matter of the identified text. ### Claim 5 (depends on 1) The method of claim 1, wherein the visually rich document comprises at least one of an invoice, a receipt, an insurance form, a boarding pass, or an identification card. ### Claim 6 A non-transitory computer-readable medium storing a plurality of instructions that when executed control a computer system to perform operations comprising: identifying, by a computing system, text in a visually rich document; determining, by the computing system, a sequence for the identified text, the sequence comprising a numerical order for each word of the identified text; selecting, by the computing system, a pretrained language model based at least in part on the identified text, the determined sequence, and a domain of the identified text; assigning, by the computing system, each word of the identified text to a respective token using the pretrained language model and the determined sequence to generate textual features corresponding to the identified text, each respective token comprising a string of one or more words; extracting, by the computing system, visual features corresponding to the identified text, the visual features comprising information about a plurality of pixels representing each word of the identified text; determining, by the computing system, positional features for each word of the identified text, the positional features comprising respective coordinates within the visually rich document for each word of the identified text, the respective coordinates for a word being coordinates corresponding to a position of the word or coordinates corresponding to a region of the visually rich document that contains the word; fusing, by the computing system, the textual features and the visual features to generate fused features; generating, by the computing system, a document model representing the visually rich document by assigning the positional features and the fused features to nodes in the document model, each node in the document model representing the fused features and the positional features of a respective word of the identified text; training, by the computing system, a classifier to classify each respective word of the identified text, the classifier being trained on the document model representing the visually rich document; and classifying, by the computing system, the respective word of the identified text with the classifier. ### Claim 7 (depends on 6) The non-transitory computer-readable medium of claim 6, wherein the respective word is classified as a key or a value of a key value pair. ### Claim 8 (depends on 6) The non-transitory computer-readable medium of claim 6, wherein the document model is a graph neural network. ### Claim 9 (depends on 6) The non-transitory computer-readable medium of claim 6, wherein the domain comprises a language or a subject matter of the identified text. ### Claim 10 (depends on 6) The non-transitory computer-readable medium of claim 6, wherein the visually rich document comprises at least one of an invoice, a receipt, an insurance form, a boarding pass, or an identification card. ### Claim 11 A system comprising: a computer-readable medium; and one or more processors for executing instructions stored on the computer-readable medium to at least perform operations comprising: identifying, by a computing system, text in a visually rich document; determining, by the computing system, a sequence for the identified text, the sequence comprising a numerical order for each word of the identified text; selecting, by the computing system, a pretrained language model based at least in part on the identified text, the determined sequence, and a domain of the identified text; assigning, by the computing system, each word of the identified text to a respective token using the pretrained language model and the determined sequence to generate textual features corresponding to the identified text, each respective token comprising a string of one or more words; extracting, by the computing system, visual features corresponding to the identified text, the visual features comprising information about a plurality of pixels representing each word of the identified text; determining, by the computing system, positional features for each word of the identified text, the positional features comprising respective coordinates within the visually rich document for each word of the identified text, the respective coordinates for a word being coordinates corresponding to a position of the word or coordinates corresponding to a region of the visually rich document that contains the word; fusing, by the computing system, the textual features and the visual features to generate fused features, generating, by the computing system, a graph representing the visually rich document by assigning the positional features and the fused features to nodes in the document model, each node in the graph representing the fused features and the positional features of a respective word of the identified text; training, by the computing system, a classifier to classify each respective word of the identified text, the classifier being trained on the graph representing the visually rich document; and classifying, by the computing system, the respective word of the identified text with the classifier. ### Claim 12 (depends on 11) The system of claim 11, wherein the respective word is classified as a key or a value of a key value pair. ### Claim 13 (depends on 11) The system of claim 11, wherein the graph is a graph neural network. ### Claim 14 (depends on 11) The system of claim 11, wherein the domain comprises a language or a subject matter of the identified text.