Representation learning of cross-language texts

An apparatus for representation learning of cross-language texts, comprising:

• at least one processor;
• and
• a memory storing computer-executable instructions that, when executed, cause the at least one processor to:
• obtain a source language text and a target language text;
• generate an initial joint representation of the source language text and the target language text;
• identify relations among a plurality of words in the source language text and the target language text;
• construct a graph corresponding to the source language text and the target language text based on the relations, wherein the graph comprises alignment edges between source language word nodes and target language word nodes and dependency edges among word nodes within a same language;
• generate a joint representation of the source language text and the target language text based on the initial joint representation and the relations by updating the initial joint representation based on the graph, wherein generating the joint representation comprises obtaining attention information based on the graph, the attention information encoding differentiated contributions from alignment neighbor nodes connected by the alignment edges and dependency neighbor nodes connected by the dependency edges, wherein a contribution from an alignment neighbor node is determined based on a semantic difference between the alignment neighbor node and a current node, and a contribution from a dependency neighbor node is determined based on an importance of the dependency neighbor node relative to the current node;
• project the joint representation to at least a target language representation corresponding to the target language text;
• and
• pre-training a representation obtaining model by:
• masking one or more nodes in a node set of a source language node set corresponding to a source language text sample and a target language node set corresponding to a target language text sample in a graph constructed based on the relations;
• for each masked node, recovering the node with at least a representation of at least one neighbor node of the node;
• computing a recovery loss based on the recovery of the masked node;
• and
• updating parameters of the representation obtaining model by minimizing the recovery loss.

At least one non-transitory machine-readable medium comprising instructions for representation learning of cross-language texts that, when executed by at least one processor, cause the at least one processor to perform operations to:

• obtaining a source language text and a target language text;
• generating an initial joint representation of the source language text and the target language text;
• identifying relations among a plurality of words in the source language text and the target language text;
• constructing a graph corresponding to the source language text and the target language text based on the relations, wherein the graph comprises alignment edges between source language word nodes and target language word nodes and dependency edges among word nodes within a same language;
• generating a joint representation of the source language text and the target language text based on the initial joint representation and the relations by updating the initial joint representation based on the graph, wherein generating the joint representation comprises obtaining attention information based on the graph, the attention information encoding differentiated contributions from alignment neighbor nodes connected by the alignment edges and dependency neighbor nodes connected by the dependency edges, wherein a contribution from an alignment neighbor node is determined based on a semantic difference between the alignment neighbor node and a current node, and a contribution from a dependency neighbor node is determined based on an importance of the dependency neighbor node relative to the current node;
• project the joint representation to at least a target language representation corresponding to the target language text;
• and
• pre-training a representation obtaining model by:
• masking one or more nodes in a node set of a source language node set corresponding to a source language text sample and a target language node set corresponding to a target language text sample in a graph constructed based on the relations;
• for each masked node, recovering the node with at least a representation of at least one neighbor node of the node;
• computing a recovery loss based on the recovery of the masked node;
• and
• updating parameters of the representation obtaining model by minimizing the recovery loss.