Transferable neural architecture for structured data extraction from web documents

A computer-implemented method of extracting data, comprising:

• generating, by one or more processors of a processing system, a document object model tree for a first page of a first website, wherein the document object model tree comprises a first plurality of nodes, and each node in the first plurality of nodes includes an XML Path (“XPath”) and content;
• generating, by the one or more processors, a prediction for each node of the first plurality of nodes regarding whether the node relates to one of a plurality of fields-of-interest;
• generating, by the one or more processors, a plurality of node-pairs from the first plurality of nodes, wherein each node-pair of the plurality comprises a head node and tail node;
• generating, by the one or more processors, a comprehensive vector corresponding to each head node and each tail node;
• generating, by the one or more processors, an XPath vector corresponding to each head node and each tail node;
• generating, by the one or more processors, a positional vector corresponding to each head node and each tail node based at least in part on a position of each head node and each tail node relative to at least one other node of the first plurality of nodes;
• for each node-pair, concatenating, by the one or more processors, the comprehensive vectors, the positional vectors, and the XPath vectors corresponding to the head node and the tail node of the node-pair to obtain a pair-level vector;
• generating, by the one or more processors, a pair label for each node-pair based on the pair-level vector for the node-pair;
• generating, by the one or more processors, for the head node of each node-pair, a node label based on the pair label for the node-pair or the prediction for the head node;
• generating, by the one or more processors, for the tail node of each node-pair, a node label based on the pair label for the node-pair or the prediction for the tail node;
• and
• extracting, by the one or more processors, structured data from one or more nodes of the first plurality of nodes, the structured data associating the content of each of the one or more nodes with the node label for each of the one or more nodes;
• wherein the comprehensive vector corresponding to each head node is based on at least a word-level vector corresponding to each word of the corresponding head node and a character-level word vector corresponding to each word of the corresponding head node;
• and
• wherein the comprehensive vector corresponding to each tail node is based on at least a word-level vector corresponding to each word of the corresponding tail node and a character-level word vector corresponding to each word of the corresponding tail node.

A processing system for extracting machine-actionable data, comprising:

• a memory;
• and
• one or more processors coupled to the memory and configured to:
• generate a document object model tree for a first page of a first website, wherein the document object model tree comprises a plurality of nodes, and each node in the plurality of nodes includes an XML Path (“XPath”) and content;
• identify a first node of the plurality of nodes, wherein the content of the first node comprises a first sequence of words, and each word in the first sequence comprises one or more characters;
• identify a second node of the plurality of nodes, wherein the content of the second node comprises a second sequence of words, each word in the second sequence comprises one or more characters, and the second sequence precedes the first sequence on the first page;
• generate a word-level vector corresponding to each word of the first sequence and the second sequence;
• generate a character-level word vector corresponding to each word of the first sequence and the second sequence;
• generate a sequence-level vector based on the word-level vectors and the character-level word vectors corresponding to the first sequence;
• generate a sequence-level vector based on the word-level vectors and the character-level word vectors corresponding to the second sequence;
• generate a discrete-features vector corresponding to one or more predefined features in the content of the first node;
• concatenate the sequence-level vector corresponding to the first sequence, the sequence-level vector corresponding to the second sequence, and the discrete-features vector to obtain a comprehensive vector for the first node;
• generate a node label for the first node based on the comprehensive vector for the first node;
• and
• extract structured data from the first node, wherein the structured data associates the content of the first node with the node label for the first node.

A processing system for extracting machine-actionable data, comprising:

• a memory;
• and
• one or more processors coupled to the memory and configured to:
• generate a document object model tree for a first page of a first website, wherein the document object model tree comprises a first plurality of nodes, and each node in the first plurality of nodes includes an XML Path (“XPath”) and content;
• generate a prediction for each node of the first plurality of nodes regarding whether the node relates to one of a plurality of fields-of-interest;
• generate a plurality of node-pairs from the first plurality of nodes, wherein each node-pair of the plurality comprises a head node and tail node;
• generate a comprehensive vector corresponding to each head node and each tail node;
• generate an XPath vector corresponding to each head node and each tail node;
• generate a positional vector corresponding to each head node and each tail node based at least in part on a position of each head node and each tail node relative to at least one other node of the first plurality of nodes;
• for each node-pair, concatenate the comprehensive vectors, the positional vectors, and the XPath vectors corresponding to the head node and the tail node of the node-pair to obtain a pair-level vector;
• generate a pair label for each node-pair based on the pair-level vector for the node-pair;
• generate, for the head node of each node-pair, a node label based on the pair label for the node-pair or the prediction for the head node;
• generate, for the tail node of each node-pair, a node label based on the pair label for the node-pair or the prediction for the tail node;
• and
• extract structured data from one or more nodes of the first plurality of nodes, wherein the structured data associates the content of each of the one or more nodes with the node label for each of the one or more nodes;
• wherein the comprehensive vector corresponding to each head node is based on at least a word-level vector corresponding to each word of the corresponding head node and a character-level word vector corresponding to each word of the corresponding head node;
• and
• wherein the comprehensive vector corresponding to each tail node is based on at least a word-level vector corresponding to each word of the corresponding tail node and a character-level word vector corresponding to each word of the corresponding tail node.