Data set generation for testing of machine learning pipelines

A computer-implemented method comprising:

• obtaining, by a computing device and from a memory, a master data set and a query expression, wherein the master data set is representable in columns and rows, and wherein the columns define fields of the master data set and the rows define entries in the master data set;
• applying, by the computing device, the query expression to the master data set to generate a test data set from the master data set, wherein applying the query expression comprises, based on content of the query expression, generating the test data set to have one or more columns or one or more rows fewer than the master data set, wherein the query expression specifies one or more columns of the master data set, one or more rows of the master data set, or a combination thereof;
• storing, by the computing device and in the memory, the test data set;
• applying, by the computing device, a machine learning (ML) pipeline to the test data set, wherein the ML pipeline includes a build determination phase and an ML model building phase, wherein the build determination phase decides whether to invoke the ML model building phase based on characteristics of an input data set, wherein the ML model building phase generates an ML model from the input data set, and wherein applying the ML pipeline results in either generation of a test ML model from the test data set or indication of an error in the test data set;
• and
• in response to applying the ML pipeline to the test data set, deleting, by the computing device, the test data set from the memory.

An article of manufacture including a non-transitory computer-readable medium, having stored thereon program instructions that, upon execution by a computing system, cause the computing system to perform operations comprising:

• obtaining, from a memory, a master data set and a query expression, wherein the master data set is representable in columns and rows, and wherein the columns define fields of the master data set and the rows define entries in the master data set;
• applying the query expression to the master data set to generate a test data set from the master data set, wherein applying the query expression comprises, based on content of the query expression, generating the test data set to have one or more columns or one or more rows fewer than the master data set, wherein the query expression specifies one or more columns of the master data set, one or more rows of the master data set, or a combination thereof;
• storing, in the memory, the test data set;
• applying a machine learning (ML) pipeline to the test data set, wherein the ML pipeline includes a build determination phase and an ML model building phase, wherein the build determination phase decides whether to invoke the ML model building phase based on characteristics of the test data set, wherein the ML model building phase generates an ML model from the test data set, and wherein applying the ML pipeline results in either generation of a test ML model from the test data set or indication of an error in the test data set;
• and
• in response to applying the ML pipeline to the test data set, deleting the test data set from the memory.