Joint Many-Task Neural Network Model for Multiple Natural Language Processing (NLP) Tasks

A multi-layer neural network system running on hardware that processes a sequence of tokens in an input sequence, the system including:

• a stacked LSTM token sequence processor, running on hardware, stacked in layers according to an analytical hierarchy, with bypass connections that deliver input to underlying layers together with embedding outputs of the underlying layers to overlaying layers, the stacked layers including:
• a first embedding layer;
• a second embedding layer overlying the first embedding layer;
• and
• a third embedding layer overlying the second embedding layer;
• the first embedding layer, implemented as a bi-directional LSTM and a first label classifier, processes token embeddings representing the tokens in the input sequence, and produces first embeddings and first state vectors of the tokens,
• the second embedding layer, implemented as a bi-directional LSTM and a second label classifier, processes at least the token embeddings, the first label embeddings and first state vectors, to produce second label embeddings and second state vectors,
• the third embedding layer, implemented as a bi-directional LSTM, processes at least the token embeddings, the first label embeddings, the second label embeddings and the second state vectors to produce third label embeddings and third state vectors, and
• an output processor that outputs at least results reflecting the third label embeddings for the tokens in the input sequence.

A method that processes words in an input sentence using a stacked layer long-short-term-memory (abbreviated LSTM) sentence processor running on hardware, stacked in layers according to a linguistic hierarchy, the stacked layers including:

• a part-of-speech (abbreviated POS) label embedding layer;
• a chunk label embedding layer overlying the POS label embedding layer;
• and
• a dependency parent identification and dependency relationship label embedding layer overlying the chunk label embedding layer;
• the method including:
• delivering, via bypass connections, input used by underlying layers together with embedding outputs from the underlying layers to overlaying layers;
• in the POS label embedding layer, applying a bi-directional LSTM and a POS label classifier to process word embedding vectors representing the words in the input sentence, and producing POS label embedding vectors and POS state vectors for each of the words;
• in the chunk label embedding layer, applying a bi-directional LSTM and a chuck label classifier to process at least the word embedding vectors, the POS label embedding vectors and the POS state vectors, and producing chunk label embeddings and chunk state vectors;
• in the dependency parent identification and dependency relationship label embedding layer, applying a bi-directional LSTM and one or more classifiers to process the word embeddings, the POS label embeddings, the chunk label embeddings and the chunk state vectors, to identify dependency parents of each of the words in the sentence and producing dependency relationship labels or label embeddings of relationships between the words and respective potential parents of the words;
• and outputting results reflecting the dependency relationship labels or label embeddings for the words in the sentence.

A method that processes tokens in an input sequence using a stacked layer long-short-term-memory (abbreviated LSTM) sentence processor running on hardware, stacked in layers according to an analytical hierarchy, the stacked layers including:

• a first embedding layer;
• a second embedding layer overlying the first embedding layer;
• and
• a third embedding layer overlying the second embedding layer;
• the method including:
• delivering, via bypass connections, input used by underlying layers together with embedding outputs of the underlying layers to overlaying layers;
• the first embedding layer, applying a bi-directional LSTM and a first label classifier to process token embeddings representing the tokens in the input sequence, and producing first label embeddings and first state vectors of the tokens;
• the second embedding layer, applying a bi-directional LSTM and a second label classifier to process at least the token embeddings, the first label embeddings and first state vectors, to produce second label embeddings and second state vectors;
• the third embedding layer, applying a bi-directional LSTM, to process at least the token embeddings, the first label embeddings, the second label embeddings and the second state vectors to produce third label embeddings and third state vectors;
• and
• outputting results reflecting stacked LSTM analysis according to the analytical hierarchy, including the third label embeddings for the tokens in the input sequence.