Compressing neural networks for natural language understanding
AssigneeOracle International Corporation
InventorMark Edward Johnson
Life of this patent
Estimating the term…
Abstract
A model for a natural language understanding task is generated based on labeled data generated by a labeling model. The model for the natural language understanding task is smaller than the labeling model (i.e., with lower computational and memory requirements than the combined model), but with substantially the same performance as the labeling model. In some cases, the labeling model may be generated based on a large pre-trained model.
Claims20
A computer-implemented method comprising:
- obtaining first labeled conversational data;
- training a first neural network using the first labeled conversational data;
- obtaining first unlabeled conversational data;
- executing the trained first neural network on the first unlabeled conversational data to label the first unlabeled conversational data, thereby generating second labeled conversational data;
- and
- training a second neural network for performing a natural language understanding task using the second labeled conversational data.
A non-transitory computer-readable memory storing a plurality of instructions executable by one or more processors, the plurality of instructions comprising instructions that when executed by the one or more processors cause the one or more processors to perform processing comprising:
- obtaining first labeled conversational data;
- training a first neural network using the first labeled conversational data;
- obtaining first unlabeled conversational data;
- executing the trained first neural network on the first unlabeled conversational data to label the first unlabeled conversational data, thereby generating second labeled conversational data;
- and
- training a second neural network for performing a natural language understanding task using the second labeled conversational data.
A system comprising:
- one or more processors;
- a memory coupled to the one or more processors, the memory storing a plurality of instructions executable by the one or more processors, the plurality of instructions comprising instructions that when executed by the one or more processors cause the one or more processors to perform processing comprising:
- obtaining first labeled conversational data;
- training a first neural network using the first labeled conversational data;
- obtaining first unlabeled conversational data;
- executing the trained first neural network on the first unlabeled conversational data to label the first unlabeled conversational data, thereby generating second labeled conversational data;
- and
- training a second neural network for performing a natural language understanding task using the second labeled conversational data.
Dependencies are inferred from the claim text.