Systems and Methods for Providing Metadata Aware Background Caching in Data Analysis

A system for providing metadata aware background caching amongst various tables in data processing systems, the system configured to process an original copy of data stored in a first format and/or data stored in one or more derived tables in one or more data stores, the system comprising:

• a cache manager, configured to copy and move data amongst various data stores, the cache manager in selective communication with the original copy of data, one or more data stores in which the one or more derived tables are stored, and/or a policy manager module;
• a policy manager module in communication with the cache manager, the policy manager comprising lifecycle policies for the original copy of data and the one or more data stores;
• and
• one or more data stores, comprising one or more derived tables that comprise data derived from the original copy of the data.

A system for providing metadata aware background caching amongst various tables in data processing systems, the system configured to process either an original copy of data stored in a first format or data stored in one or more derived tables in one or more data stores, the system comprising:

• a query optimization module comprising a cost-based optimizer configured to determine a most desirable manner of conducting queries, and further configured to conduct queries against data stored in the original copy of data and/or in the one or more derived tables;
• a catalog module configured to perform metadata reads of each of the original copy of the data and the one or more derived tables, and further configured to register tables of data across various types and formats of data stores;
• a dataset manager configured to maintain the freshness of the data in the one or more derived tables, the data set manager comprising:
• an event listener module, the event listener module configured to initiate a data manipulation language (DML) operation when prompted;
• a scheduler module, configured to regularly and/or periodically check if policies associated with the original copy of the data and the one or more derived tables are maintained;
• and
• an executor module, configured to submit DML commands.