System and method for monitoring machine learning models
AssigneeRoyal Bank of Canada
InventorLeandro Axel Guelman
Life of this patent
Estimating the term…
Abstract
Systems and methods are provided to monitor performance of a machine learning model, the method may include steps of: receiving or storing one or more model data sets representative of the machine learning model, wherein the machine learning model has being trained with a first set of training data; analyzing the first set of training data based on one or more performance parameters for the machine learning model, to generate one or more performance data sets; and process the one or more performance data sets to determine one or more values representing a performance of the machine learning model.
Claims19
A computer implemented system for monitoring and improving a performance of one or more machine learning models, the system including:
- at least one memory storage device storing one or more model data sets representative of a machine learning model;
- at least one training engine configured to train the machine learning model;
- and
- at least one computer processor configured to, when executing a set of machine-readable instructions:
- receive or store the one or more model data sets representative of the machine learning model, wherein the machine learning model has being trained with a first set of training data;
- analyze the first set of training data, based on one or more performance parameters for the machine learning model, to generate one or more performance data sets, wherein the one or more performance parameters comprises a population stability index value configured to assess a stability of one or more output data sets of the machine learning model;
- process the one or more performance data sets to determine one or more values representing a performance of the machine learning model;
- select a second set of training data based on the one or more performance data sets;
- re-train the machine learning model using the second set of training data;
- and
- display, at a display device, the one or more values representing the performance of the machine learning model, including displaying the population stability index value at the display device in response to a user input;
- wherein the population stability index value is generated based on a J-divergence measure and is configured to assess the stability of the one or more output data sets based on a pre-determined threshold value.
The system of 1, wherein the one or more performance parameters comprises at least one of:
- a regression feature and a classification feature of the machine learning model.
A computer-implemented method for monitoring and improving a performance of a machine learning model, the method comprising:
- receiving or storing one or more model data sets representative of the machine learning model, wherein the machine learning model has being trained with a first set of training data;
- analyzing the first set of training data, based on one or more performance parameters for the machine learning model, to generate one or more performance data sets, wherein the one or more performance parameters comprises a population stability index value configured to assess a stability of one or more output data sets of the machine learning model;
- processing the one or more performance data sets to determine one or more values representing a performance of the machine learning model;
- selecting a second set of training data based on the one or more performance data sets;
- re-training the machine learning model using the second set of training data;
- and
- displaying, at a display device, the one or more values representing the performance of the machine learning model, including displaying the population stability index value at the display device in response to a user input;
- wherein the population stability index value is generated based on a J-divergence measure and is configured to assess the stability of the one or more output data sets based on a pre-determined threshold value.
The method of 12, wherein the one or more performance parameters comprises at least one of:
- a regression feature and a classification feature of the machine learning model.
A computer implemented system for determining an output based on a set of input using a machine learning model, the system including:
- at least one memory storage device storing one or more model data sets representative of a machine learning model;
- and
- at least one computer processor configured to, when executing a set of machine-readable instructions, execute the one or more model data sets representative of the machine learning model to generate an output based on a set of input data,
- wherein the machine learning model has being trained with a first set of training data, and wherein the processor is configured to:
- analyze the first set of training data, based on one or more performance parameters for the machine learning model, to generate one or more performance data sets, wherein the one or more performance parameters comprises a population stability index value configured to assess a stability of one or more output data sets of the machine learning model;
- process the one or more performance data sets to determine one or more values representing a performance of the machine learning model;
- select a second set of training data based on the one or more performance data sets;
- re-train the machine learning model using the second set of training data;
- and
- display, at a display device, the one or more values representing the performance of the machine learning model, including displaying the population stability index value at the display device in response to a user input;
- wherein the population stability index value is generated based on a J-divergence measure and is configured to assess the stability of the one or more output data sets based on a pre-determined threshold value.
Dependencies are inferred from the claim text.