System and method for explaining and compressing deep learning natural language understanding (NLU) models
AssigneeSamsung Electronics Co., Ltd.
InventorsYilin Shen, Hongxia Jin
Life of this patent
Estimating the term…
Abstract
A method includes obtaining, using at least one processor of an electronic device, a base natural language understanding (NLU) model that includes a word embedding layer, where the word embedding layer is associated with at least one training utterance. The method also includes calculating, using the at least one processor, a regularization loss value for use in a determination of an intent detection loss, where the regularization loss value reveals an effect of word embeddings on intent determination of the training utterance. The method further includes retraining, using the at least one processor, the word embedding layer of the base NLU model using the intent detection loss to obtain a retrained NLU model.
Claims20
A method comprising:
- obtaining, using at least one processor of an electronic device, a base natural language understanding (NLU) model comprising a word embedding layer, the word embedding layer associated with at least one training utterance;
- calculating, using the at least one processor, a regularization loss value, wherein the regularization loss value reveals an effect of word embeddings on intent determination of the at least one training utterance;
- determining, using the at least one processor, a regularization coefficient by (i) adding a plurality of pseudo words to the at least one training utterance and (ii) performing an iterative process in which the base NLU model is trained in multiple iterations, each iteration associated with an updated value for the regularization coefficient;
- determining, using the at least one processor, an intent detection loss based on a cross entropy loss, the regularization coefficient, and the regularization loss value;
- and
- retraining, using the at least one processor, the word embedding layer of the base NLU model using the intent detection loss to obtain a retrained NLU model.
An electronic device comprising:
- at least one memory configured to store instructions;
- and
- at least one processing device configured when executing the instructions to:
- obtain a base natural language understanding (NLU) model comprising a word embedding layer, the word embedding layer associated with at least one training utterance;
- calculate a regularization loss value, wherein the regularization loss value reveals an effect of word embeddings on intent determination of the at least one training utterance;
- determine a regularization coefficient by (i) adding a plurality of pseudo words to the at least one training utterance and (ii) performing an iterative process in which the base NLU model is trained in multiple iterations, each iteration associated with an updated value for the regularization coefficient;
- determine an intent detection loss based on a cross entropy loss, the regularization coefficient, and the regularization loss value;
- and
- retrain the word embedding layer of the base NLU model using the intent detection loss to obtain a retrained NLU model.
A non-transitory machine-readable medium containing instructions that when executed cause at least one processor of an electronic device to:
- obtain a base natural language understanding (NLU) model comprising a word embedding layer, the word embedding layer associated with at least one training utterance;
- calculate a regularization loss value, wherein the regularization loss value reveals an effect of word embeddings on intent determination of the at least one training utterance;
- determine a regularization coefficient by (i) adding a plurality of pseudo words to the at least one training utterance and (ii) performing an iterative process in which the base NLU model is trained in multiple iterations, each iteration associated with an updated value for the regularization coefficient;
- determine an intent detection loss based on a cross entropy loss, the regularization coefficient, and the regularization loss value;
- and
- retrain the word embedding layer of the base NLU model using the intent detection loss to obtain a retrained NLU model.
Dependencies are inferred from the claim text.