Method and apparatus for application of an N-dimensional hypercube datatype

A system for application of an N-dimensional hypercube datatype for automatically generating a data structure, the system comprising:

• a database that stores a plurality of data each associated with a corresponding application and each including metadata, wherein the metadata describes information about data present in an application, information about data that an application is authoritative, and information about data that flows between applications;
• and
• a processor coupled to the database via a communication network, wherein the processor is configured to:
• create taxonomies describing data concepts associated with the metadata and store the taxonomies onto the database;
• receive the metadata and the taxonomies describing the data concepts from the database via a communication network;
• automatically generate a cube set including a set of N-dimensional hypercubes from the received metadata to represent the metadata describing the information about the data present in the application, the information about the data identifying that the application is authoritative, and the information about the data that flows between applications, wherein the set of N-dimensional hypercubes forms a data model for generating a data structure;
• automatically generate a lineage map from the cube set to express data quality checks and rules that apply to nodes in the lineage map;
• automatically check whether a node in the lineage map is acting as a system of record (SOR) or as an authoritative data source (ADS) for the same set of data, wherein:
• when the node creates data, that node is referred to as the SOR for that data,
• when the node does not receive data, but sends the data that it creates to another node of the lineage map for further processing, then recording that the node is acting as the SOR for that data,
• when a node receives data and sends the data to another node of the lineage map for further processing, then recording that the node is acting as the ADS for that data, and
• the lineage map includes metadata corresponding to one or more of the following metadata:
• calculated SOR, calculated ADS, declared SOR, declared ADS, calculated SOR minus declared SOR, calculated ADS minus declared ADS, declared SOR minus calculated SOR, declared ADS minus calculated ADS, and overlap of ADS and SOR;
• apply the data quality checks and the rules to the received metadata to automatically generate the data structure from the received metadata to represent the set of N-dimensional hypercubes as the data model;
• apply the data structure to model data in various nodes of the data model and data flowing between those nodes to support calculation and reasoning about data present in the nodes and data flowing between them;
• automatically detect, based on the lineage map, that data leaving the application is not more than data entering the application including any data created by the application, thereby improving accuracy in the metadata recorded;
• and
• automatically detect and resolve data quality issues based on the lineage map thereby reducing size of data for analysis and storage.

A non-transitory computer readable medium configured to store instructions for applying an N-dimensional hypercube datatype for automatically generating a data structure, wherein, when executed, the instructions cause a processor to perform the following:

• creating taxonomies describing data concepts associated with metadata and storing the taxonomies onto a database;
• accessing the database, wherein the database stores a plurality of data each associated with a corresponding application and each including the metadata, and wherein the metadata describes information about data present in an application, information about data that an application is authoritative, and information about data that flows between applications;
• receiving the metadata and the taxonomies describing the data concepts from the database;
• automatically generating a cube set including a set of N-dimensional hypercubes from the received metadata to represent the metadata describing the information about the data present in the application, the information about the data identifying that the application is authoritative, and the information about the data that flows between applications, wherein the set of N-dimensional hypercubes forms a data model for generating a data structure;
• automatically generating a lineage map from the cube set to express data quality checks and rules that apply to nodes in the lineage map;
• and
• automatically checking whether a node in the lineage map is acting as a system of record (SOR) or as an authoritative data source (ADS) for the same set of data, wherein:
• when the node creates data, that node is referred to as the SOR for that data,
• when the node does not receive data, but sends the data that it creates to another node of the lineage map for further processing, then recording that the node is acting as the SOR for that data,
• when a node receives data and sends the data to another node of the lineage map for further processing, then recording that the node is acting as the ADS for that data, and
• the lineage map includes metadata corresponding to one or more of the following metadata:
• calculated SOR, calculated ADS, declared SOR, declared ADS, calculated SOR minus declared SOR, calculated ADS minus declared ADS, declared SOR minus calculated SOR, declared ADS minus calculated ADS, and overlap of ADS and SOR;
• applying the data quality checks and the rules to the received metadata to automatically generate the data structure from the received metadata to represent the set of N-dimensional hypercubes as the data model;
• applying the data structure to model data in various nodes of the data model and data flowing between those nodes to support calculation and reasoning about data present in the nodes and data flowing between them;
• and
• automatically detecting, based on the lineage map, that data leaving the application is not more than data entering the application including any data created by the application, thereby improving accuracy in the metadata recorded;
• and
• automatically detecting and resolving data quality issues based on the lineage map thereby reducing size of data for analysis and storage.