System and method for determination of model fitness and stability for model deployment in automated model generation

A method for determination of model fitness and stability for model deployment in automated model generation, comprising:

• providing a computer comprising one or more microprocessors, and a data analytics environment operating thereon;
• wherein the data analytics environment operating on the computer includes one or more software components operating as a data plane, and providing access to one or more data warehouse instances;
• wherein the data plane includes a data pipeline or process that maintains, for each of a plurality of tenants, a data analytics schema that is updated on a periodic basis by the system;
• wherein the data plane includes a data transformation layer that the system uses to transform transactional data into a model format understood by the data analytics environment;
• and
• wherein the data analytics environment provides, for each tenant of the plurality of tenants, a customer schema which allows the tenant to supplement and utilize data within their own data warehouse instance, so that their resultant data warehouse instance operates as a database whose contents are partly-controlled by the tenant, and partly-controlled by the data analytics environment;
• providing, at the data analytics environment operating on the computer, a model fitness and stability component and a plurality of models;
• based upon a set of data at the data analytics environment, scoring a set of the plurality of data models, wherein scoring each data model within the set comprises:
• automatically assigning predictions of the data model to probability bins, and
• determining a successive differencing of correct classifications between successive probability bins, for use in determining a probability bins score;
• selecting, based upon the scoring, a selected data model from within the set of the plurality of models;
• and
• monitoring the selected data model for indications of instability or drift.

A non-transitory computer readable storage medium, including instructions stored thereon which when read and executed by one or more computers cause the one or more computers to perform a method comprising:

• providing, at a computer comprising one or more microprocessors, a data analytics environment operating thereon;
• wherein the data analytics environment operating on the computer includes one or more software components operating as a data plane, and providing access to one or more data warehouse instances;
• wherein the data plane includes a data pipeline or process that maintains, for each of a plurality of tenants, a data analytics schema that is updated on a periodic basis by the system;
• wherein the data plane includes a data transformation layer that the system uses to transform transactional data into a model format understood by the data analytics environment;
• and
• wherein the data analytics environment provides, for each tenant of the plurality of tenants, a customer schema which allows the tenant to supplement and utilize data within their own data warehouse instance, so that their resultant data warehouse instance operates as a database whose contents are partly-controlled by the tenant, and partly-controlled by the data analytics environment;
• providing, at the data analytics environment operating on the computer, a model fitness and stability component and a plurality of models;
• based upon a set of data at the data analytics environment, scoring a set of the plurality of data models, wherein scoring each data model within the set comprises:
• automatically assigning predictions of the data model to probability bins, and
• determining a successive differencing of correct classifications between successive probability bins, for use in determining a probability bins score;
• selecting, based upon the scoring, a selected data model from within the set of the plurality of models;
• and
• monitoring the selected data model for indications of instability or drift.