Training systems for pseudo labeling natural language
AssigneeCapital One Services, LLC
InventorsAditya Relangi, Michael Langford
Life of this patent
Estimating the term…
Abstract
Examples of the present disclosure can comprise systems and methods for creating and modifying named entity recognition models. The system can use two or more existing named entity recognition models to output responses to natural language queries for which the models have not yet been trained. When the output from the two or more models match, the query and the resulting output can be stored as training data for a new named entity recognition model. If the output from the two or models do not match, the query and the outputs can be stored in an exceptions file for additional review. In some embodiments, the system can comprise one or more processors and a display for providing a user interface (UI).
Claims20
A system for providing natural language processing and interactive responses, the system comprising:
- one or more processors;
- a display for providing a user interface (UI), the UI comprising:
- an input field for receiving an input by a user;
- and
- a display window for displaying the user input and one or more responses;
- a natural language processing (NLP) device comprising two or more trained named entity recognition models;
- and
- a memory in communication with at least the one or more processors, the display, and the NLP device, the memory storing instructions that, when executed by the one or more processors, are configured to cause the one or more processors to:
- apply, using the two or more trained named entity recognition models, pseudo labels to named entities in a first natural language request;
- receive, from each of the two or more trained named entity recognition models, a response, each response comprising two or more named entities identified in the first natural language request with corresponding pseudo labels;
- determine that the responses from the two or more trained named entity recognition models do not match;
- display, on the user interface (UI), a notification to a reviewer, the notification comprising the two or more responses and a confidence level associated with each of the two or more trained named entity recognition models;
- receive, via the UI, a corrected response from the reviewer;
- store, in the memory, the corrected response in a training data set;
- and
- train a first named entity recognition model using the training data set.
A system for providing natural language processing and interactive responses, the system comprising:
- a display comprising a user interface (UI), the UI including an input field for receiving input from a user;
- one or more processors;
- a natural language processing (NLP) device comprising two or more trained named entity recognition models;
- and
- a memory in communication with at least the one or more processors, the display, and the NLP device, the memory storing instructions that, when executed by the one or more processors, are configured to cause the system to:
- train a first named entity recognition model using a training data set, the training data set comprising previous natural language requests labeled by the two or more trained named entity recognition models;
- add the first named entity recognition model to the two or more trained named entity recognition models;
- apply, using the two or more trained named entity recognition models, pseudo labels to named entities in a first natural language request;
- receive, from each of the two or more trained named entity recognition models, a response, each response comprising two or more named entities identified in the first natural language request with corresponding pseudo labels;
- determine that the responses from the two or more trained named entity recognition models do not match;
- provide the first natural language request and the two or more responses to a second named entity recognition model that is separate from the two or more trained named entity recognition models;
- and
- receive, from the second named entity recognition model, a corrected response to the request;
- store, in the memory, the corrected response in a training data set;
- and
- train a third named entity recognition model using the training data set.
A method for providing natural language processing and interactive responses, the method comprising:
- receiving, via an input field on a user interface (UI), a first natural language request;
- providing, to a natural language processing (NLP) device comprising two or more trained named entity recognition models, the first natural language request;
- processing, using the two or more trained named entity recognition models, the first natural language request to apply pseudo labels to named entities in the first natural language request;
- receiving, from each of the two or more trained entity recognition models, a response, each response comprising two or more named entities with corresponding pseudo labels;
- determining that the two or more responses do not match;
- storing, in a memory, the first natural language request and the two or more responses in an entry in an exceptions file;
- displaying, on a user interface (UI), a notification to a reviewer to review the exceptions file;
- receiving, via the UI, a corrected response from the review;
- storing the corrected response in a new training data set;
- and
- training a first named entity recognition model using the new training data set.
Dependencies are inferred from the claim text.