US 2018/0121799 A1 · Training a Joint Many-Task Neural Network Model using Successive Regularization

Abstract

The technology disclosed provides a so-called “joint many-task neural network model” to solve a variety of increasingly complex natural language processing (NLP) tasks using growing depth of layers in a single end-to-end model. The model is successively trained by considering linguistic hierarchies, directly connecting word representations to all model layers, explicitly using predictions in lower tasks, and applying a so-called “successive regularization” technique to prevent catastrophic forgetting. Three examples of lower level model layers are part-of-speech (POS) tagging layer, chunking layer, and dependency parsing layer. Two examples of higher level model layers are semantic relatedness layer and textual entailment layer. The model achieves the state-of-the-art results on chunking, dependency parsing, semantic relatedness and textual entailment.

Claims22

Dependent claims

A method of training a stacked LSTM sequence processor, running on hardware, stacked in at least three layers according to an analytical hierarchy, the training method including:

training first, second and third layers by backward propagation using training examples directed to each layer, with regularized pass down to underlying layers during training, including:
training the first layer using first layer training examples,
training the second layer using second layer training examples with regularized pass down training to the first layer, and
training the third layer using third layer training examples with regularized pass down training to the first and second layers;
and
regularized pass down training is regularized by constraining a training objective function, having a fitness function with at least two regularization terms;
and
the two regularization terms regularize by penalizing growth in a magnitude of weights in coefficient matrices applied to the underlying layers and that successively regularize all changes in the weights in the coefficient matrices applied to the underlying layers.

5 more elements

A method of conveying intermediate results from an underlying layer to an overlying layer in a neural network stack of bidirectional LSTMs, in which the stack has layers corresponding to an analytical framework that process a sequence of tokens, and the underlying layer produces analytic framework label vectors for each of the tokens, the method including:

for the sequence, analyzing the tokens using the underlying layer, including:
applying the bidirectional LSTM to compute forward and backward state vectors for each of the tokens;
applying a classifier to the forward and backward state vectors to embed each of the tokens in an analytic framework label space, as label space vectors that have dimensionality about the same as a number of available analytical framework labels;
and
projecting the label space vectors of each token into an extended dimensionality label space, which has dimensionality about the same as dimensionality of the forward and backward states, to produce extended token label vectors;
and
conveying, from the underlying layer to the overlying layer, vectors of the forward state, the backward state, and the extended token label, thereby supplying input needed by the overlying layer to perform its role in the analytical framework for processing the tokens.

4 more elements

A multi-layer neural network system running on hardware that processes a sequence of tokens in an input sequence, the system including:

a stacked LSTM token sequence processor, running on hardware, stacked in layers according to an analytical hierarchy, with bypass connections that deliver input to underlying layers together with embedding outputs of the underlying layers to overlaying layers, the stacked layers including:
a first embedding layer;
and
a second embedding layer overlying the first embedding layer;
the first embedding layer, implemented as a bi-directional LSTM and a first label classifier, processes token embeddings representing the tokens in the input sequence, and produces analytic framework label vectors for each of the tokens, further including:
the bi-directional LSTM computes forward and backward state vectors for each of the tokens;
a classifier applied to the forward and backward state vectors embeds each of the tokens in an analytic framework label space, as label space vectors that have dimensionality about the same as a number of available analytical framework labels;
and
the first embedding layer projects the label space vectors of each token into an extended dimensionality label space, which has dimensionality about the same as dimensionality of the forward and backward states, to produce extended token label vectors;
and
the first embedding layer sending to the second embedding layer, vectors of the forward state, the backward state, and the extended token label, thereby supplying input needed by the second embedding layer to perform its role in the analytical framework for processing the tokens.

8 more elements

Dependencies are inferred from the claim text.

Training a Joint Many-Task Neural Network Model using Successive Regularization

Life of this patent

Training a Joint Many-Task Neural Network Model using Successive Regularization

Life of this patent

Abstract

Claims22