System and methods for performing nlp related tasks using contextualized word representations

A system for improving the performance of a neural network used for a natural language understanding (NLU) or a natural language processing (NLP) task, comprising:

• a set of computer-executable instructions for representing a natural language sequence or sequences with a bidirectional language model;
• a set of computer-executable instructions for implementing the bidirectional language model in a neural network;
• a set of computer-executable instructions for training the neural network in which the language model is implemented using a corpus of unlabeled text;
• a set of computer-executable instructions for extracting contextual word representations from the trained neural network, the contextual word representations being extracted from one or more layers of the trained network;
• and
• a set of computer-executable instructions for transferring the extracted contextual word representations to the neural network used for the natural language understanding (NLU) or natural language processing (NLP) task.

A method for improving the performance of a neural network used for a natural language understanding (NLU) or a natural language processing (NLP) task, comprising:

• representing a sequence of tokens as a bidirectional language model;
• implementing the bidirectional language model in the form of a neural network;
• training the neural network in which the language model is implemented using a corpus of text;
• identifying representations for each token in the sequence from one or more layers of the resultant trained neural network;
• forming a contextual vector or vectors for each token from the identified representations of the trained neural network;
• obtaining a neural network intended for use in a specific task;
• introducing the contextual vectors into the obtained neural network;
• inputting text to be analyzed into the obtained neural network;
• and
• generating an output of the obtained neural network.