Tracking provenance in data science scripts

A machine learning data provenance tracking system, comprising:

• a processor;
• a derivation extractor component executed by the processor, the derivation extractor component being configured to receive machine learning (ML) model code, the ML model code using at least a first ML application programming interface (API) that when executed, generates and trains an ML model, the derivation extractor component further configured to determine a workflow intermediate representation (WIR) of the ML model code;
• a queryable knowledge base (KB) stored in one or more memory devices, the queryable KB including:
• a plurality of ML API annotations corresponding to the semantic identity of, and relationships between, inputs and outputs of operations provided by a plurality of ML APIs, including the first ML API;
• and
• a table of operations corresponding to ML model code operations that perform data selection or exclusion operations;
• a ML model code analyzer component executed by the processor, the ML model code analyzer being configured to semantically annotate the WIR based at least in part on the first ML API associated with the ML model code to generate an annotated WIR;
• and
• a provenance tracker component executed by the processor, the provenance tracker component being configured to identify, based at least on the annotated WIR and KB query results, data from at least one data source that is relied upon by the ML model code when training the ML model.

A computer-readable storage medium having program instructions recorded thereon that, when executed by at least one processor, perform a method, the method comprising:

• receiving machine learning (ML) model code, the ML model code using at least a first ML application programming interface (API) that when executed, generates and trains an ML model;
• determining a workflow intermediate representation (WIR) of the ML model code by:
• building an abstract syntax tree (AST) based at least in part on the received ML model code;
• generating a plurality of provenance relationships (PRs) based at least in part on relationships between nodes of the AST, wherein a PR comprises one or more input variables, an operation, a caller, and one or more output variables, wherein said generating the plurality of PRs comprises:
• traversing the AST starting at each root node of the AST and for each node of the AST that is not a literal or constant, determining an operation corresponding to the node, and recursively determining each one or more inputs, a caller and one or more outputs corresponding to the node, wherein the determined operation, one or more inputs, caller and one or more outputs together comprise a generated PR;
• and
• composing a directed graph based at least in part on the plurality of PRs, wherein the WIR comprises the directed graph;
• semantically annotating the WIR based at least in part on the first ML API associated with the ML model code to generate an annotated WIR;
• and
• identifying, based at least on the annotated WIR and the first ML API, data from at least one data source that is relied upon by the ML model code when training the ML model.

The computer-readable storage medium of 15 wherein the semantically annotating the WIR based at least in part on the first ML API associated with the ML model code to generate the annotated WIR comprises:

• accessing a queryable knowledge base (KB) that includes ML API annotations corresponding to a semantic identity of, and relationships between, inputs and outputs of operations provided by a plurality of ML APIs, including the first ML API;
• and
• adding annotations to the PRs of the directed graph corresponding to the WIR by, beginning at each import process node of the directed graph, performing a forward traversal of PRs in the directed graph, and for each PR encountered in the forward direction (forward PR):
• querying the queryable KB for first input annotations corresponding to the one or more input variables of the forward PR, for first output annotations corresponding to the one or more output variables of the forward PR, and adding the first input annotations and first output annotations to the forward PR;
• and
• for each of the first input annotations, performing a backward traversal of PRs in the directed graph starting with the forward PR, and for each PR encountered in the backward direction (backward PR):
• querying the queryable KB based on the first input annotations for second input annotations corresponding to the backward PR, and adding the second input annotations to the backward PR.