Hybrid approach for measuring statistical drift and data quality on large datasets

One or more storage media storing instructions which, when executed by one or more computing devices, cause:

• accessing a data monitoring definition that includes predefined configuration data and a custom configuration data that is specified by a user, wherein the custom configuration data includes custom instructions pertaining to one or more of data reading, metrics generation, or data writing;
• based on the data monitoring definition, generating executable code that comprises a data reading portion, a metrics generation portion, and a data writing portion;
• executing the executable code, wherein executing the executable code comprises:
• based on the data reading portion in the executable code, reading a dataset based on location data specified in the monitoring definition;
• based on the metrics generation portion in the executable code, generating a set of metrics based on the dataset;
• based on the data writing portion in the executable code, writing a result that is based on the set of metrics.

A system comprising:

• one or more computing devices;
• one or more storage media storing instructions which, when executed by the one or more computing devices, cause:
• accessing a data monitoring definition that includes predefined configuration data and a custom configuration data that is specified by a user, wherein the custom configuration data includes custom instructions pertaining to one or more of data reading, metrics generation, or data writing;
• based on the data monitoring definition, generating executable code that comprises a data reading portion, a metrics generation portion, and a data writing portion;
• executing the executable code, wherein executing the executable code comprises:
• based on the data reading portion in the executable code, reading a dataset based on location data specified in the monitoring definition;
• based on the metrics generation portion in the executable code, generating a set of metrics based on the dataset;
• based on the data writing portion in the executable code, writing a result that is based on the set of metrics.