Methods for unsupervised prediction of performance drop due to domain shift
AssigneeNaver France
InventorsMatthias GALLE, Hady ELSAHAR
Life of this patent
Estimating the term…
Abstract
A system includes: a natural language processing (NLP) model trained in a training domain and configured to perform natural language processing on an input dataset; an accuracy module configured to: calculate a domain shift metric based on the input dataset; and calculate a predicted decrease in accuracy of the NLP model attributable to domain shift relative to the training domain based on the domain shift metric; and a retraining module configured to selectively trigger a retraining of the NLP model based on the predicted decrease in accuracy of the NLP model.
Claims20
A system comprising:
- a natural language processing (NLP) model trained in a training domain and configured to perform natural language processing on an input dataset;
- an accuracy module configured to:
- calculate a domain shift metric based on the input dataset;
- and
- calculate a predicted decrease in accuracy of the NLP model attributable to domain shift relative to the training domain based on the domain shift metric;
- and
- a retraining module configured to selectively trigger a retraining of the NLP model based on the predicted decrease in accuracy of the NLP model.
A system comprising:
- a training module configured to:
- train, using a training dataset in a training domain, a machine learning (ML) model to perform processing on an input dataset;
- determine properties of a domain shift metric for the training dataset in the training domain;
- an accuracy module configured to:
- calculate the domain shift metric based on the input dataset;
- and
- calculate a predicted decrease in accuracy of the ML model attributable to domain shift relative to the training domain based on the domain shift metric using the properties of the domain shift metric for the training dataset;
- and
- a retraining module configured to selectively trigger a retraining of the ML model based on the predicted decrease in accuracy of the ML model.
A computerized method performed by one or more processors, the method comprising:
- training a natural language processing (NLP) model using a training dataset to perform natural language processing on an input dataset;
- after receiving the input dataset that is different than the training dataset:
- calculating a domain shift metric based on the input dataset;
- and
- calculating a predicted decrease in accuracy of the NLP model attributable to domain shift relative to a training domain based on the domain shift metric;
- and
- selectively triggering a retraining of the NLP model based on the predicted decrease in accuracy of the NLP model.
A system comprising:
- a natural language processing (NLP) model trained in a training domain and configured to perform natural language processing on an input dataset;
- a means for:
- calculating a domain shift metric based on the input dataset;
- and
- calculating a predicted decrease in accuracy of the NLP model attributable to domain shift relative to the training domain based on the domain shift metric;
- and
- a means for selectively trigger a retraining of the NLP model based on the predicted decrease in accuracy of the NLP model.
Dependencies are inferred from the claim text.