Structural encoding and attention paradigms for sequence modeling

A processing system comprising:

• a memory storing a neural network comprising a transformer;
• and
• one or more processors coupled to the memory and configured to classify text from a given document, comprising:
• generating a beta-skeleton graph based on a plurality of tokens, wherein the beta-skeleton graph comprises, for each given token of the plurality of tokens:
• a node corresponding to the given token and comprising a vector based on content and location of a given string of text within the given document;
• and
• one or more edges, each edge of the one or more edges linking the node corresponding to the given token to a neighboring node corresponding to another token of the plurality of tokens;
• and
• an edge embedding for each edge based on a spatial relationship comprising at least one of vertical distance, horizontal distance, or aspect ratio between the tokens;
• generating, using a graph convolutional network, a plurality of supertokens based on the beta-skeleton graph, wherein the plurality of supertokens preserves local syntactic information independently of document serialization, each given supertoken of the plurality of supertokens being based at least in part on the vector of a given node and the vector of each neighboring node to which the given node is linked via one of its one or more edges
• generating, using a transformer, a plurality of predictions based on the plurality of tokens, each given token of the plurality of tokens corresponding to the given string of text in the given document, and wherein generating a given prediction of the plurality of predictions for a given attender token and a given attendee token of the plurality of tokens comprises:
• generating a first prediction regarding how the given attender token and given attendee token should be ordered if the given attender token and given attendee token are related to one another;
• generating a second prediction regarding how far the given attender token should be from the given attendee token if the given attender token and given attendee token are related to one another;
• generating a first error value based on the first prediction and a value based on how the text corresponding to the given attender token and given attendee token is actually ordered in the given document;
• generating a second error value based on the second prediction and a value based on how far the text corresponding to the given attender token actually is from the text corresponding to the given attendee token in the given document;
• generating a query vector based on the given attender token;
• generating a key vector based on the given attendee token;
• generating a first attention score based on the query vector and the key vector;
• generating a second attention score based on the first attention score, the first error value, and the second error value;
• and
• generating the given prediction based at least in part on the second attention score;
• and
• generating a set of classifications based on the plurality of predictions, the set of classifications identifying at least one entity class corresponding to at least one token of the plurality of tokens.