US 2023/0144585 A1Published application

Machine learning model change detection and versioning

AssigneeInternational Business Machines Corporation

InventorsShubhi Asthana, Shikhar Kwatra, Sushain Pandit

Life of this patent

Estimating the term…

Loading patent…

Abstract

Systems, methods, and computer programming products for versioning machine learning models. Changes between new and existing datasets are detected, quantified and compared using statistical and semantic feature comparisons. Recommendations for versioning existing models are in response to detecting changes between the feature importance of datasets used in the application of the machine learning model and new datasets that introduce new features or features that evolve over time in such a manner that feature importance has shifted away from one or more features of the first dataset to the new dataset. Based on the changes in feature importance, statistical changes and semantic feature comparisons, the recommendations provided describe whether models should be updated with a re-trained model, or that the existing features of the model do not indicate a need for re-training.

Claims20

Dependent claims

A computer-implemented method for versioning a machine learning model, the computer-implemented method comprising:

ingesting, by a versioning service, a first dataset configured to train the machine learning model;
performing, by the versioning service, feature exploration of the first data set and extracting from the first dataset, feature importance (f1) of the machine learning model;
ranking, by the versioning service, top features of the first dataset used to train the machine learning model by the feature importance, up to a configured threshold number (n) of features;
pre-processing, by the versioning service, features of a second dataset (f2);
comparing, by the versioning service, changes in features between f1 and f2 for up to the configured threshold number of features;
and
upon comparing, by the versioning service, the changes in the features between f1 and f2, and the changes between f1 and f2 are non-overlapping features:
highlighting set (f1−f2) in f1 which have an addition or deletion of categories within a feature and if the set (f1−f2) is ranked within the top features up to the configured threshold number of features for f1, outputting, by the versioning service, a recommendation to re-train the machine learning model.

5 more elements

A computer program product for versioning a machine learning model comprising:

one or more computer readable storage media having computer-readable program instructions stored on the one or more computer readable storage media, said program instructions executes a computer-implemented method comprising:
ingesting, by a versioning service, a first dataset configured to train the machine learning model;
performing, by the versioning service, feature exploration of the first data set and extracting from the first dataset, feature importance (f1) of the machine learning model;
ranking, by the versioning service, top features of the first dataset used to train the machine learning model by the feature importance, up to a configured threshold number (n) of features;
pre-processing, by the versioning service, features of a second dataset (f2);
comparing, by the versioning service, changes in features between f1 and f2 for up to the configured threshold number of features;
and
upon comparing, by the versioning service, the changes in the features between f1 and f2, and the changes between f1 and f2 are non-overlapping features:
highlighting set (f1−f2) in f1 which have an addition or deletion of categories within a feature and if the set (f1−f2) is ranked within the top features up to the configured threshold number of features for f1, outputting, by the versioning service, a recommendation to re-train the machine learning model.

6 more elements

A computer system for versioning a machine learning model comprising:

a processor;
and
a computer-readable storage media coupled to the processor, wherein the computer-readable storage media contains program instructions executing a computer-implemented method comprising:
ingesting, by a versioning service, a first dataset configured to train the machine learning model;
performing, by the versioning service, feature exploration of the first data set and extracting from the first dataset, feature importance (f1) of the machine learning model;
ranking, by the versioning service, top features of the first dataset used to train the machine learning model by the feature importance, up to a configured threshold number (n) of features;
pre-processing, by the versioning service, features of a second dataset (f2);
comparing, by the versioning service, changes in features between f1 and f2 for up to the configured threshold number of features;
and
upon comparing, by the versioning service, the changes in the features between f1 and f2, and the changes between f1 and f2 are non-overlapping features:
highlighting set (f1−f2) in f1 which have an addition or deletion of categories within a feature and if the set (f1−f2) is ranked within the top features up to the configured threshold number of features for f1, outputting, by the versioning service, a recommendation to re-train the machine learning model.

8 more elements

Dependencies are inferred from the claim text.