US 12,718,136 B1Granted

System for automatic predictive model training and release

AssigneeWorkday, Inc.

InventorsYon-Seo Kim, Parag Avinash Namjoshi, Joseph Turian, Yury Markovsky, Lakshminarayanan Renganarayana

Life of this patent

Estimating the term…

Loading patent…

Abstract

A system for automatic model training and release includes a model storage system, an interface, and a processor. The interface is configured to receive an indication to create a model for release. The processor is configured to fetch data for creation of the model; validate the data to determine a data validation result; extract features based at least on part the data; validate the features using one or more feature quality metrics; build the model based at least in part on the features and the data; validate the model using one or more model quality metrics; store the model using the model storage system; and release the model for use.

Claims20

Dependent claims

A system for automatic model training and release, comprising:

a tenanted area of a multitenanted server system, wherein the tenanted area comprises (i) a tenanted data storage storing data only accessible to a tenant and not other tenants and (ii) a model storage system storing one or more models only accessible to the tenant and not the other tenants;
an interface configured to:
receive an indication to create a model for release;
and
a processor configured to:
fetch data for creation of the model from the tenanted data storage, wherein the fetched data is associated with the tenant;
validate the fetched data to determine a data validation result;
extract features based at least in part on the fetched data;
validate the features using one or more feature quality metrics;
build the model based at least in part on the features and the fetched data;
validate the model using one or more metrics;
store the model in the model storage system storing the one or more models only accessible to the tenant and not the other tenants;
release the model for use by the tenant as a released model, wherein releasing the model comprises providing access to the released model for use by the tenant without providing access to the released model for use by the other tenants;
after providing access to the released model for use by the tenant, determine whether the released model fails validation based at least in part on a comparison between a second set of time series tenant data and an output of the released model executed using a first set of time series tenant data;
in response to a determination that the released model fails the validation, determine a number of times that the released model has failed validation and determine whether the number of times that the released model has failed validation at least meets a predetermined specific number of times, the predetermined specific number of times being greater than one;
and
in response to a determination that the number of times that the released model has failed validation at least meets the predetermined specific number of times, remove access to the released model for use by the tenant.

14 more elements

A method for automatic model training and release, comprising:

receiving an indication to create a model for release;
fetching, using a processor, data for creation of the model from a tenanted data storage storing data only accessible to a tenant and not other tenants, wherein a tenanted area of a multitenanted server system comprises the tenanted data storage, and wherein the fetched data is associated with the tenant;
validating the fetched data using to determine a data validation result;
extracting features based at least in part on the fetched data;
validating the features using one or more metrics;
building the model based at least in part on the features and the fetched data;
validating the model using one or more metrics;
storing the model in a model storage system storing one or more models only accessible to the tenant and not the other tenants, wherein the tenanted area of the multitenanted server system comprises the model storage system;
releasing the model for use by the tenant as a released model, wherein releasing the model comprises providing access to the released model for use by the tenant without providing access to the released model for use by the other tenants;
after providing access to the released model for use by the tenant, determining that the released model fails validation based at least in part on a comparison between a second set of time series tenant data and an output of the released model executed using a first set of time series tenant data;
in response to a determination that the released model fails the validation, determining a number of times that the released model has failed validation and determine whether the number of times that the released model has failed validation at least meets a predetermined specific number of times, the predetermined specific number of times being greater than one;
and
in response to a determination that the number of times that the released model has failed validation at least meets the predetermined specific number of times, removing access to the released model for use by the tenant.

10 more elements

A computer program product for automatic model training and release, the computer program product being embodied in a non-transitory computer readable storage medium and comprising computer instructions for:

receiving an indication to create a model for release;
fetching data for creation of the model from a tenanted data storage storing data only accessible to a tenant and not other tenants, wherein a tenanted area of a multitenanted server system comprises the tenanted data storage, and wherein the fetched data is associated with the tenant;
validating the fetched data to determine a data validation result;
extracting features based at least in part on the fetched data;
validating the features using one or more feature quality metrics;
building the model based at least in part on the features and the fetched data;
validating the model using one or more metrics;
storing the model in a model storage system storing one or more models only accessible to the tenant and not the other tenants, wherein the tenanted area of the multitenanted server system comprises the model storage system;
releasing the model for use by the tenant as a released model, wherein releasing the model comprises providing access to the released model for use by the tenant without providing access to the released model for use by the other tenants;
after providing access to the released model for use by the tenant, determining whether the released model fails validation based at least in part on a comparison between a second set of time series tenant data and an output of the released model executed using a first set of time series tenant data;
in response to a determination that the released model fails the validation, determining a number of times that the released model has failed validation and determine whether the number of times that the released model has failed validation at least meets a predetermined specific number of times, the predetermined specific number of times being greater than one;
and
in response to a determination that the number of times that the released model has failed validation at least meets the predetermined specific number of times, removing access to the released model for use by the tenant.

10 more elements

Dependencies are inferred from the claim text.