Data revision control in large-scale data analytic systems

One or more non-transitory computer-readable media storing instructions for execution by one or more processors, the instructions configured for performing operations comprising:

• storing a first version of a first dataset that is derived from a first version of a second dataset based on a first execution of a first version of a driver program;
• storing, in a build catalog, a first build catalog entry comprising an identifier of the first version of the first dataset, an identifier of the first version of the second dataset, a first branch name, and an identifier of the first version of the driver program;
• storing a second version of the first dataset that is derived from a second version of the second dataset based on a second execution of the first version of the driver program;
• storing, in the build catalog, a second build catalog entry comprising an identifier of the second version of the first dataset, an identifier of the second version of the second dataset, a second branch name that is different from the first branch name, and an identifier of the first version of the driver program, the build catalog including the first build catalog entry and the second build catalog entry;
• and
• causing display of a provenance graph in a graphical user interface based on the first build catalog entry and the second build catalog entry, the provenance graph display comprising:
• a first node representing the first version of the first dataset, a second node representing the first version of the second dataset, a third node representing the second version of the first dataset, a fourth node, representing the second version of the second dataset, a first directed edge from the first node to the second node, and a second directed edge from the third node to the fourth node.

A system, comprising:

• one or more processors;
• one or more storage media storing one or more programs which, when executed by the one or more processors, cause:
• storing a first version of a first dataset that is derived from a first version of a second dataset based on a first execution of a first version of a driver program;
• storing, in a build catalog, a first build catalog entry comprising an identifier of the first version of the first dataset, an identifier of the first version of the second dataset, a first branch name, and an identifier of the first version of the driver program;
• storing a second version of the first dataset that is derived from a second version of the second dataset based on a second execution of the first version of the driver program;
• storing, in the build catalog, a second build catalog entry comprising an identifier of the second version of the first dataset, an identifier of the second version of the second dataset, a second branch name that is different from the first branch name, and an identifier of the first version of the driver program, the build catalog including the first build catalog entry and the second build catalog entry;
• and
• causing display of a provenance graph in a graphical user interface based on the first build catalog entry and the second build catalog entry, the provenance graph display comprising:
• a first node representing the first version of the first dataset, a second node representing the first version of the second dataset, a third node representing the second version of the first dataset, a fourth node, representing the second version of the second dataset, a first directed edge from the first node to the second node, and a second directed edge from the third node to the fourth node.