Deep Neural Network Model for Processing Data Through Mutliple Linguistic Task Hiearchies

A dependency parsing layer component of a neural network system, running on hardware, that processes words in an input sentence;

• the dependency parsing layer overlies a chunk label embedding layer that produces chunk label embeddings and chunk state vectors;
• the chunk label embedding layer, in turn, overlies a POS label embedding layer that produces POS label embeddings and POS state vectors;
• the device component including:
• a dependency parent layer and a dependency relationship label classifier;
• the dependency parent layer includes:
• a dependency parent analyzer, implemented as a bi-directional LSTM, that processes the words in the input sentences, including processing, for each word, word embeddings, the POS label embeddings, the chunk label embeddings, and the chunk state vector to accumulate forward and backward state vectors that represent forward and backward progressions of interactions among the words in the sentence;
• and
• an attention encoder that processes the forward and backward state vectors for each respective word in the sentence, and encodes attention as inner products between each respective word and other words in the sentence, with a linear transform applied to the forward and backward state vectors for the word or the other words prior to the inner product;
• the attention encoder further applies exponential normalization to vectors of the inner products to produce parent label probability mass vectors and projects the parent label probability mass vectors to produce parent label embedding vectors;
• and
• the dependency relationship label classifier, for each respective word in the sentence:
• processes embedding vectors and the parent label embedding vectors, to produce dependency relationship label probability mass vectors;
• and
• projects the dependency relationship label probability mass vectors to produce dependency relationship label embedding vectors;
• and
• an output processor that outputs at least results reflecting classification labels for the dependency relationship of each word, the dependency relationship label probability mass vectors, or the dependency relationship label embedding vectors.

A method of dependency parsing using a neural network device, running on hardware, that processes words in an input sentence;

• a dependency parsing layer overlies a chunk label embedding layer that produces chunk label embeddings and chunk state vectors;
• the chunk label embedding layer, in turn, overlies a POS label embedding layer that produces POS label embeddings;
• the dependency parsing layer includes a dependency parent layer and a dependency relationship label classifier;
• the method including:
• in the dependency parent layer:
• applying a bi-directional LSTM and one or more classifiers, that process the word embeddings, the POS label embeddings, the chunk label embeddings and the chunk state vectors, to produce parent label probability mass vectors by classification and exponential normalization of parent label state vectors produced by the bi-directional LSTM;
• and
• producing the parent label embedding vectors from the parent label probability mass vectors;
• and
• in the dependency relationship label classifier:
• producing dependency relationship label probability mass vectors by classification and exponential normalization of the parent label state vectors and the parent label embedding vectors;
• and
• producing the dependency relationship label embedding vectors from the dependency relationship label probability mass vectors;
• and
• wherein dimensionalities of the POS label embedding vectors, the chunk label embedding vectors, and the dependency relationship label embedding vectors are similar, within +/−ten percent;
• and
• further including at least the dependency relationship label embedding vectors or dependency relationship labels based thereon.

A method of dependency parsing using a neural network device, running on hardware, that processes words in an input sentence;

• a dependency parsing layer overlies a chunk label embedding layer that produces chunk label embeddings and chunk state vectors;
• the chunk label embedding layer, in turn, overlies a POS label embedding layer that produces POS label embeddings;
• the dependency parsing layer includes a dependency parent layer and a dependency relationship label classifier;
• the method including:
• in the dependency parent layer:
• in a dependency parent analyzer, applying a bi-directional LSTM to process the words in the input sentences, including processing, for each word, word embeddings, the POS label embeddings, the chunk label embeddings, and the chunk state vector to accumulate forward and backward state vectors that represent forward and backward progressions of interactions among the words in the sentence;
• and
• in an attention encoder that processes the forward and backward state vectors for each respective word in the sentence;
• encoding attention as inner products between each respective word and other words in the sentence, with a linear transform applied to the forward and backward state vectors for the word or the other words prior to the inner product;
• applying exponential normalization to vectors of the inner products to produce parent label probability mass vectors and projects the parent label probability mass vectors to produce parent label embedding vectors;
• and
• in the dependency relationship label classifier, for each respective word in the sentence:
• classifying and normalizing the forward and backward state vectors and the parent label embedding vectors and the parent label embedding vectors, to produce dependency relationship label probability mass vectors;
• and
• projecting the dependency relationship label probability mass vectors to produce dependency relationship label embedding vectors;
• and
• outputting at least results reflecting classification labels for the dependency relationship of each word, the dependency relationship label probability mass vectors, or the dependency relationship label embedding vectors.

A method of preparing words in an input sentence, including words not previously mapped into a word embedding space, for processing by multi-layer neural network system running on hardware, using a word embedder and a substring embedder, both of which process the words in the input sentence, the method including:

• in the word embedder, mapping previously recognized words into a word embedding space and identifying previously unrecognized words, to produce word embedding vectors for each of the words;
• in the substring embedder, for each of the words in the input sentence:
• processing character substrings of the word at multiple scales of substring length;
• mapping each processed character substring into an intermediate vector representing a position in a character embedding space;
• and
• combining the intermediate vectors for each unique processed character substring to produce character embedding vectors for each of the words;
• and
• outputting the word embedding vectors and the character embedding vectors, whereby a word not previously mapped into the word embedding space is nonetheless represented by the character embedding vector.

A multi-layer neural network system running on hardware that processes words in an input sentence, including words not previously mapped into a word embedding space, the system including:

• a word embedder and a substring embedder, both of which process the words in the input sentence;
• the word embedder maps previously recognized words into a word embedding space and identifies previously unrecognized words, to produce word embedding vectors for each of the words;
• the substring embedder that
• processes character substrings of the word at multiple scales of substring length;
• maps each processed character substring into an intermediate vector representing a position in a character embedding space;
• and
• combines the intermediate vectors for each unique processed character substring to produce character embedding vectors for each of the words;
• and
• an embedder combiner that reports for use by a further process or processing layer both the word embedding vectors and the character embedding vectors, whereby a word not previously mapped into the word embedding space is nonetheless represented by the character embedding vector.