Transactional version sets

#Today, many users and organizations build data lakes to break down data silos, centralize data assets, and unlock the potential of their data. Many of these users use data lake management services to simplify data lake setup, security, and management. For example, users may use services that can model their data objects as tables with semi-structured data, centrally define security policies on tables and their columns, and securely analyze the data through a wide variety of analytics and ML services.

#As these users look to leverage data lakes for an increasing variety of purposes, they want granular control over how data is updated and accessed—like they have with data warehouses today. For example, to load streaming data or quickly incorporate changes from source data systems, customers need to insert, delete, and modify rows across multiple tables in parallel. At the same time, to support real-time analytics and ML, they need to run queries that return consistent and up-to-date data. To meet these needs today, users write and maintain complex and error-prone application code to manage concurrent updates and ensure queries return accurate results. Existing systems do not support transactions across multiple tables, have difficulty scaling, and may only work with a narrow set of open-source analytics engines.

#Moreover, many users seek to perform “time travel” based analysis of their data. For example, time travel is increasingly an important part of the data governance and compliance needs for many organizations. Time travel enables accessing historical data as of a specific time, within some defined window. Time travel is typically required to support audits (in case of audits customers often need to run/rerun a report as of a specific time), restoring data (in case of accidental or malicious deletions), backup (to create snapshots/backups for a specific time such as midnight every night), etc.

#Various embodiments in accordance with the present disclosure will be described with reference to the drawings, in which:

#FIG. 1 is a diagram illustrating an environment including a transactional access manager of a data lake management system for implementing a governed storage interface providing transactional support for concurrent modifications to objects in a data lake according to some embodiments.

#FIG. 2 is a diagram illustrating exemplary data structures utilized by a transactional access manager for enabling transactional data lake access with time travel capability according to some embodiments.

#The present disclosure relates to methods, apparatus, systems, and non-transitory computer-readable storage media for implementing transactional support for data stored in systems such as data lakes. According to some embodiments, a transactional version set can be used to serve as a fundamental abstraction, e.g., for implementing catalog metadata of various types. Data structures following the transactional version set organization can be efficiently stored, provide fast access to requested data, and allow for potentially parallel transactional access to the data with atomicity while ensuring isolation, optionally avoiding the need for locking. The transactional version set, in some embodiments, can be implemented in a hierarchical manner to provide a single representation used for multiple types of catalog entities and operations, simplifying the logic and complexity of a system. In some embodiments, transactional version sets can be used to implement a consistent data catalog, which maintains metadata about the databases, tables, partitions, and files that make up a distributed data management system such as a data lake.

#In some embodiments, transactional version set inspired organizations can be utilized with a protocol for implementing atomic, consistent, isolated, and durable (ACID) transactions that provide snapshot isolation in a distributed setting. Snapshot isolation is commonly used in databases, but most implementations rely on a centralized clock or sequence number generator, which can become a bottleneck in distributed environments. In some embodiments, a protocol and implementation may not require synchronized clocks, e.g., by ensuring at commit time that transactions touching common objects do not commit out of order. In some embodiments, this protocol is implemented in the context of a distributed data lake built on an object storage service in which clients can transactionally add or remove objects from logical tables.

#In some embodiments, an architecture constructed using transactional version set concepts may include the use of tree data structures, such as B-Trees, to maintain a transactionally consistent set with support for time-travel queries. In some embodiments, when a transaction commits, a new version of a tree is created using a copy-on-write based approach that shares internal nodes (with trees corresponding to earlier versions) to save space. In some embodiments, this approach is implemented in a transactional data catalog in which the files that make up a table are stored in a transactional set.