Change control and version management of data

A system comprising:

• at least one hardware processor;
• and
• at least one non-transitory memory storing instructions, which, when executed by the at least one hardware processor, cause the system to:
• perform a change to a data lake within a commit,
• wherein the commit defines a set of changes to the data lake that are applied as a single atomic operation;
• define multiple branches that each define a separate chain of commits for multiple data objects,
• wherein a new commit updates a head of a branch associated with the new commit;
• create an indication that defines a static reference to the commit of the data lake;
• merge a source commit onto a target commit including merging a head of one branch onto the head of another branch;
• configure different versions of a particular data object of the multiple data objects to grant different privileges to different entities;
• and
• configure each version of the particular data object such that privileges are global across all versions of the particular data object on each branch of the multiple branches,
• wherein an access to a particular version of the particular data object on a particular branch of the multiple branches requires the grant of privileges on the particular data object and privileges on the particular branch,
• wherein the privileges on the particular data object apply to all versions of the particular data object on each branch of the multiple branches such that the privileges on the particular data object allow access to new versions of the particular data object on a new branch, and
• wherein the privileges on the particular branch include privileges to access the particular branch for the particular version of the particular data object.

A method for performing distributed version control on a data lake built on a large scale-out storage system, the method comprising:

• performing a change to the data lake within a single atomic commit that defines a set of changes to the data lake,
• wherein the change includes a modification to the data lake including adding data to a dataset, creating a dataset, or modifying a dataset,
• wherein a subsequent commit is built on top of another commit and describes incremental changes, which creates a sequence of atomic commits that defines a history of changes, and
• wherein each commit is identified by a hash value;
• defining multiple branches as references to commits for multiple data objects,
• wherein a new commit updates a branch head reference to the new commit such that that the branch head always refers to a latest version of a chain of commits in the branch;
• creating an indication that defines a static reference for the data lake;
• merging a source commit onto a target commit including merging the head of one branch onto the head of another branch,
• wherein the source branch includes a series of commits that are replayed on top of the target branch in an order on the source branch;
• configuring different versions of a particular data object of the multiple data objects to grant different privileges to different entities;
• configuring each version of the particular data object such that privileges are global across all versions of the particular data object on each branch of the multiple branches,
• wherein an access to a particular version of the particular data object on a particular branch of the multiple branches requires the grant of privileges on the particular data object and privileges on a particular branch,
• wherein the privileges on the particular data object apply to all versions of the particular data object on each branch of the multiple branches such that the privileges on the particular data object allow access to new versions of the particular data object on a new branch, and
• wherein the privileges on the particular branch include privileges to access the particular branch for the particular version of the particular data object.