Domain adaptation of AI NLP encoders with knowledge distillation
AssigneeLiveperson, Inc.
InventorsKristen Howell, Jian Wang, Matthew Dunn, Joseph Bradley
Life of this patent
Estimating the term…
Abstract
Systems, methods, devices, instructions, and other examples are described for natural language processing. One example includes accessing natural language processing general encoder data, where the encoder data is generated from a general-domain dataset that is not domain specific. A domain specific dataset is accessed and filtered encoder data using a subset of the encoder data is generated. The filtered encoder data is trained using the domain specific dataset to generate distilled encoder data, and tuning values for the distilled encoder data are generated to configure task outputs associated with the domain specific dataset.
Claims20
A computer-implemented method comprising:
- accessing natural language processing (NLP) general encoder data, wherein the NLP general encoder data is generated from a general-domain dataset that is not domain specific, and wherein the NLP general encoder data includes layers;
- generating filtered encoder data by removing a subset of the layers of the NLP general encoder data;
- training the filtered encoder data using a domain specific dataset to generate distilled encoder data;
- and
- analyzing, using the distilled encoder data, datasets for domain specific NLP.
A device comprising:
- a memory storing instructions;
- and
- one or more processors coupled to the memory, the one or more processors configured by the instructions to perform operations comprising:
- accessing natural language processing (NLP) general encoder data, wherein the NLP general encoder data is generated from a general-domain dataset that is not domain specific, and wherein the NLP general encoder data includes layers;
- generating filtered encoder data by removing a subset of the layers of the NLP general encoder data;
- training the filtered encoder data using a domain specific dataset to generate distilled encoder data;
- and
- analyzing, using the distilled encoder data, datasets for domain specific NLP.
A non-transitory computer readable storage medium comprising instructions that, when executed by one or more processors of a device, cause the device to perform operations comprising:
- accessing natural language processing (NLP) general encoder data, wherein the NLP general encoder data is generated from a general-domain dataset that is not domain specific, and wherein the NLP general encoder data includes layers;
- generating filtered encoder data by removing a subset of the layers of the NLP general encoder data;
- training the filtered encoder data using a domain specific dataset to generate distilled encoder data;
- and
- analyzing, using the distilled encoder data, datasets for domain specific NLP.
Dependencies are inferred from the claim text.