Executing unsupervised pre-training tasks with a machine learning model to predict document graph attributes

A method comprising:

• executing an unsupervised first training stage for training a machine learning model to predict document graph attributes at least by:
• accessing a set of unlabeled training data comprising a first set of document graphs corresponding to a first set of documents;
• wherein a first document graph in the first set of document graphs comprises:
• a first node corresponding to a first grammatical element comprising a plurality of tokens in a first document of the first set of documents, the first node comprising first text information for the first grammatical element in the first document and at least one of first positional information representing a first position of a first text content in a first document and first font style data for the first text content;
• a second node corresponding to a second grammatical element in the first document, the second node comprising second text information for the second grammatical element in the first document and at least one of second positional information representing a second position of a second text content in the first document and second font style data for the second text content;
• and
• a first edge connecting the first node and the second node, the first edge comprising relationship information reflecting a relationship of the first node to the second node,
• wherein the relationship information includes at least one of:
• a read-order relationship of the first text content and the second text content;
• a coordinate location relationship of the first text content and the second text content;
• and
• a relationship of the first font style data to the second font style data;
• generating a training dataset for the unsupervised first training stage of the machine learning model by generating a set of modified document graph data based on the first set of document graphs, wherein generating the training dataset for the unsupervised first training stage of the machine learning model comprises:
• executing a graph-data-modification process to modify document graph attributes of at least the first node to generate a first set of modified document graph data corresponding to the first document, the first set of modified document graph data including the modified document graph attributes of the first node;
• and
• training the machine learning model with the training dataset by iteratively applying the machine learning model to document graphs in the training dataset to predict the modified document graph attributes, at least by:
• applying the machine learning model to a particular document graph in the training dataset to predict a particular document attribute of the particular document, wherein the particular document attribute corresponds to the modified document graph data;
• comparing the prediction by the machine learning model to an unmodified document graph corresponding to the particular document graph to determine whether the prediction is correct;
• and
• updating parameters of the machine learning model based on results of comparing the prediction to the unmodified document graph.

A system comprising:

• one or more processors;
• and
• memory storing instructions that, when executed by the one or more processors, cause the system to perform:
• executing an unsupervised first training stage for training a machine learning model to predict document graph attributes at least by:
• accessing a set of unlabeled training data comprising a first set of document graphs corresponding to a first set of documents;
• wherein a first document graph in the first set of document graphs comprises:
• a first node corresponding to a first grammatical element comprising a plurality of tokens in a first document of the first set of documents, the first node comprising first text information for the first grammatical element in the first document and at least one of first positional information representing a first position of a first text content in a first document and first font style data for the first text content;
• a second node corresponding to a second grammatical element in the first document, the second node comprising second text information for the second grammatical element in the first document and at least one of second positional information representing a second position of a second text content in the first document and second font style data for the second text content;
• and
• a first edge connecting the first node and the second node, the first edge comprising relationship information reflecting a relationship of the first node to the second node,
• wherein the relationship information includes at least one of:
• a read-order relationship of the first text content and the second text content;
• a coordinate location relationship of the first text content and the second text content;
• and
• a relationship of the first font style data to the second font style data;
• generating a training dataset for the unsupervised first training stage of the machine learning model by generating a set of modified document graph data based on the first set of document graphs, wherein generating the training dataset for the unsupervised first training stage of the machine learning model comprises:
• executing a graph-data-modification process to modify document graph attributes of at least the first node to generate a first set of modified document graph data corresponding to the first document, the first set of modified document graph data including the modified document graph attributes of the first node;
• and
• training the machine learning model with the training dataset by iteratively applying the machine learning model to document graphs in the training dataset to predict the modified document graph attributes, at least by:
• applying the machine learning model to a particular document graph in the training dataset to predict a particular document attribute of the particular document, wherein the particular document attribute corresponds to the modified document graph data;
• comparing the prediction by the machine learning model to an unmodified document graph corresponding to the particular document graph to determine whether the prediction is correct;
• and
• updating parameters of the machine learning model based on results of comparing the prediction to the unmodified document graph.