Domain adapting graph networks for visually rich documents

A non-transitory computer-readable medium storing a plurality of instructions that when executed control a computer system to perform operations comprising:

• identifying, by a computing system, text in a visually rich document;
• determining, by the computing system, a sequence for the identified text, the sequence comprising a numerical order for each word of the identified text;
• selecting, by the computing system, a pretrained language model based at least in part on the identified text, the determined sequence, and a domain of the identified text;
• assigning, by the computing system, each word of the identified text to a respective token using the pretrained language model and the determined sequence to generate textual features corresponding to the identified text, each respective token comprising a string of one or more words;
• extracting, by the computing system, visual features corresponding to the identified text, the visual features comprising information about a plurality of pixels representing each word of the identified text;
• determining, by the computing system, positional features for each word of the identified text, the positional features comprising respective coordinates within the visually rich document for each word of the identified text, the respective coordinates for a word being coordinates corresponding to a position of the word or coordinates corresponding to a region of the visually rich document that contains the word;
• fusing, by the computing system, the textual features and the visual features to generate fused features;
• generating, by the computing system, a document model representing the visually rich document by assigning the positional features and the fused features to nodes in the document model, each node in the document model representing the fused features and the positional features of a respective word of the identified text;
• training, by the computing system, a classifier to classify each respective word of the identified text, the classifier being trained on the document model representing the visually rich document;
• and
• classifying, by the computing system, the respective word of the identified text with the classifier.

A system comprising:

• a computer-readable medium;
• and
• one or more processors for executing instructions stored on the computer-readable medium to at least perform operations comprising:
• identifying, by a computing system, text in a visually rich document;
• determining, by the computing system, a sequence for the identified text, the sequence comprising a numerical order for each word of the identified text;
• selecting, by the computing system, a pretrained language model based at least in part on the identified text, the determined sequence, and a domain of the identified text;
• assigning, by the computing system, each word of the identified text to a respective token using the pretrained language model and the determined sequence to generate textual features corresponding to the identified text, each respective token comprising a string of one or more words;
• extracting, by the computing system, visual features corresponding to the identified text, the visual features comprising information about a plurality of pixels representing each word of the identified text;
• determining, by the computing system, positional features for each word of the identified text, the positional features comprising respective coordinates within the visually rich document for each word of the identified text, the respective coordinates for a word being coordinates corresponding to a position of the word or coordinates corresponding to a region of the visually rich document that contains the word;
• fusing, by the computing system, the textual features and the visual features to generate fused features,
• generating, by the computing system, a graph representing the visually rich document by assigning the positional features and the fused features to nodes in the document model, each node in the graph representing the fused features and the positional features of a respective word of the identified text;
• training, by the computing system, a classifier to classify each respective word of the identified text, the classifier being trained on the graph representing the visually rich document;
• and
• classifying, by the computing system, the respective word of the identified text with the classifier.