Distributed and interactive database architecture for parallel and asynchronous data processing of complex data and for real-time query processing

A data processing system, comprising:

• a control interface to receive a first plurality of user-modifiable and dynamically selectable data processing parameters, a second plurality of user-modifiable and dynamically selectable data processing parameters, and a third plurality of user-modifiable and dynamically selectable data processing parameters, the control interface further to modify the first, second and third pluralities of selectable data processing parameters in response to received, user-modifiable and dynamically selectable control information determining data process values, data attributes, data confidence levels, and data process ordering;
• a data input to receive input data;
• a data and messaging network coupled to the control interface and to the data input interface;
• a first data repository coupled to the data and messaging network, the first data repository to store a plurality of entity data and a plurality of corresponding entity attribute data for a plurality of people and households;
• a second, linkage data repository coupled to the data and messaging network, the second data repository to store a plurality of unique and persistent personal identifiers wherein each unique and persistent personal identifier corresponds to each unique person or household of the plurality of people and households and further to store a plurality of corresponding, comparative confidence levels for matching personal data;
• a control processor to control the performance of a plurality of data processes in the selected order with the selected data process values, data attributes and data confidence levels;
• a data assembly processor coupled to the data and messaging network, the data assembly processor to perform a data assembly process of the plurality of data processes using the first plurality of selectable data processing parameters to generate a first plurality of data from the first data repository, from the second data repository, and from input data;
• a third data repository coupled to the data and messaging network, the third data repository to store the first plurality of data;
• a data reduction processor coupled to the data and messaging network, the data reduction processor to perform a data reduction process of the plurality of data processes using the second plurality of selectable data processing parameters to generate a second plurality of data from the first plurality of data;
• a fourth data repository coupled to the data and messaging network, the fourth data repository to store the second plurality of data;
• an aggregation processor coupled to the data and messaging network, the aggregation processor to perform a data aggregation process of the plurality of data processes using the third plurality of selectable data processing parameters to dimension and aggregate the second plurality of data, to determine a plurality of sets of unique and persistent personal identifiers from a multidimensional data structure and to perform a plurality of set operations on the plurality of sets of unique and persistent personal identifiers;
• a fifth data repository coupled to the data and messaging network, the fifth data repository having the multidimensional data structure to store the dimensioned and aggregated second plurality of data;
• and
• a sixth, metadata repository coupled to the data and messaging network, the sixth, metadata repository to store a plurality of metadata modules defining a plurality of data structures stored in the first, second, third, fourth and fifth data repositories, defining a marketing campaign, and further defining the plurality of data processes and selectable data process values, data attributes, data confidence levels, and data process ordering.

A parallel and asynchronous data processing system for marketing campaign management, comprising:

• a user interface;
• a control interface to receive user-modifiable and dynamically selectable control information determining a plurality of data process values, data attributes, data confidence levels, and data process ordering;
• a plurality of data processing nodes coupled through a data and messaging network to the user interface and to the control interface, the plurality of data processing nodes to perform a plurality of data processes;
• a control processor to control the performance of the plurality of data processes in the selected order with the selected data process values, data attributes and data confidence levels;
• a first data repository coupled through the data and messaging network to the plurality of data processing nodes, the first data repository to store a plurality of entity attribute information for a plurality of people and groups of people;
• a linkage data repository coupled through the data and messaging network to the plurality of data processing nodes, the linkage data repository to store a plurality of unique and persistent personal identifiers wherein each unique and persistent identifier corresponds to each unique person or group of people of the plurality of people and groups of people and further to store a plurality of corresponding, comparative confidence levels for matching personal data;
• a second data repository coupled through the data and messaging network to the plurality of data processing nodes, the second data repository to store a first subset of information from the first data repository and the linkage data repository, the first subset of information including a first subset of entity attribute information;
• a metadata repository coupled through the data and messaging network to the plurality of data processing nodes, the metadata repository to store a plurality of metadata modules defining a plurality of data structures stored in the first, linkage and second data repositories, defining a marketing campaign, and further defining the plurality of data processes and selectable data process values, data attributes, data confidence levels, and data processing orders;
• a data storage system coupled through the data and messaging network to the plurality of data processing nodes, the data storage system to store a multidimensional data structure, the multidimensional data structure having an aggregation of the first subset of information dimensioned with a first plurality of selected attributes of the first subset of entity attribute information and stored as a corresponding first subset of the plurality of unique and persistent personal identifiers, wherein the first plurality of selected attributes are modifiable as selectable data processing parameters of metadata during data processing through the user interface or the control interface;
• wherein at least one first processing node of the plurality of processing nodes further is to determine the first subset of information stored in the second data repository and to dimension and aggregate the first subset of information using the first plurality of selected attributes;
• and
• wherein at least one second processing node of the plurality of processing nodes further is to determine a plurality of sets of unique and persistent personal identifiers from the multidimensional data structure and to perform a plurality of set operations on the plurality of sets of unique and persistent personal identifiers.

A data processing method for marketing campaign management, comprising:

• storing a plurality of entity and attribute data in a first data repository of a plurality of data repositories;
• storing a plurality of entity linkage data in a second data repository of the plurality of data repositories, the plurality of entity linkage data comprising a plurality of unique and persistent personal identifiers corresponding to a plurality of people and households and further comprising a plurality of corresponding, comparative confidence levels for matching personal data;
• receiving a plurality of user-modifiable and dynamically selectable data processing parameters determining data process values, data attributes, data confidence levels, and data process ordering;
• storing a plurality of metadata modules in a metadata data repository of the plurality of data repositories, the plurality of metadata modules defining a plurality of data structures stored in the first and second data repositories, defining a marketing campaign, and further defining a plurality of data processes and selectable data processing parameters defining data process values, data attributes, data confidence levels, and data process ordering, a first metadata module of the plurality of metadata modules referencing the plurality of selectable data processing parameters;
• controlling the performance of the plurality of data processes in the selected order with the selected data process values, data attributes and data confidence levels;
• using the plurality of selectable data processing parameters, assembling a first plurality of data from the first and second data repositories and from input data;
• using the plurality of selectable data processing parameters, reducing the first plurality of data to form a second plurality of data;
• using the plurality of selectable data processing parameters, dimensioning and aggregating the second plurality of data;
• storing the aggregated and dimensioned second plurality of data in a multidimensional data structure;
• determining a plurality of sets of unique and persistent personal identifiers from the multidimensional data structure;
• and
• performing a plurality of set operations on the plurality of sets of unique and persistent personal identifiers.

A computer readable storage medium storing computer readable software for programming a parallel and asynchronous database architecture and data processing system for execution of marketing campaign management and analysis, the computer readable storage medium storing computer readable software comprising:

• a first program module to receive a plurality of user-modifiable and dynamically selectable data processing parameters determining data process values, data attributes, data confidence levels, and data process ordering, to modify the plurality of selectable data processing parameters in response to received control information or in response to modeled information to form a modified plurality of selectable data processing parameters, and to control the performance of a plurality of data processes in the selected order with the selected data process values, data attributes and data confidence levels;
• a second program module to store a plurality of entity and attribute data in a first data repository of a plurality of data repositories and to store a plurality of entity linkage data in a second data repository of the plurality of data repositories, the plurality of entity linkage data comprising a plurality of unique and persistent personal identifiers corresponding to a plurality of people and households and further comprising a plurality of corresponding, comparative confidence levels for matching personal data;
• and to store a plurality of metadata modules in a metadata data repository of the plurality of data repositories, the plurality of metadata modules defining a plurality of data structures stored in the first and second data repositories, defining a marketing campaign, and further defining a plurality of data processes and selectable data processing parameters defining data process values, data attributes, data confidence levels, and data process ordering, a first metadata module of the plurality of metadata modules referencing the plurality of selectable data processing parameters;
• a third program module to use the plurality of selectable data processing parameters to assemble in parallel and asynchronously a first plurality of data from the first and second data repositories and from input data;
• to reduce the first plurality of data to form a second plurality of data;
• and to dimension and aggregate the second plurality of data;
• a fourth program module to store the dimensioned and aggregated second plurality of data in a multidimensional data structure as a corresponding set of unique and persistent personal identifiers of the plurality of unique and persistent personal identifiers;
• a fifth program module to process a query and provide a query response using the multidimensional data structure;
• a sixth program module to use the modified plurality of selectable data processing parameters to reduce the first plurality of data to form a modified second plurality of data;
• and to use the modified plurality of selectable data processing parameters to dimension and aggregate the modified second plurality of data;
• and
• a seventh program module to perform a plurality of set operations on the plurality of sets of unique and persistent personal identifiers.

A data processing system for marketing campaign management, comprising:

• a plurality of data repositories having a corresponding plurality of data structures, a first data repository of the plurality of data repositories to store a plurality of entity data and entity attribute data, a second data repository of the plurality of data repositories to store a plurality of entity linkage data comprising a plurality of unique and persistent personal identifiers corresponding to a plurality of people and further comprising a plurality of corresponding, comparative confidence levels for matching personal data, a third data repository of the plurality of data repositories to store a plurality of metadata modules, wherein the plurality of metadata modules define a plurality of data processes, define a marketing campaign, and further define the plurality of data structures and a multidimensional data structure, and wherein a first metadata module of the plurality of metadata modules comprises a plurality of selectable data processing parameters determining data process values, data attributes, data confidence levels, and data process ordering;
• a control interface coupled to the plurality of data repositories, the control interface further comprising a user interface to select the plurality of selectable data processing parameters, to select input data sources, to select confidence levels for data matching, to select a plurality of attributes for data processing, to select and order the performance of a subset of data processes of the plurality of data processes, and to select a plurality of dimensions for aggregation;
• a data storage system storing the multidimensional data structure;
• a plurality of processing nodes coupled to the plurality of data repositories, to the control interface, and to the data storage system, the plurality of processing nodes to perform the subset of data processes in the selected order using the plurality of selectable data processing parameters determining data process values, data attributes, data confidence levels, and data process ordering, using the selection of the plurality of attributes, and using the selection of the plurality of dimensions, wherein the subset of data processes comprises assembling asynchronously and in parallel a first plurality of data from the first and second data repositories and from the selection of input data sources, asynchronously reducing the first plurality of data to form a second plurality of data, and dimensioning and aggregating the second plurality of data for storage as a set of unique and persistent personal identifiers, of the plurality of unique and persistent personal identifiers, in the multidimensional data structure in the data storage device;
• and
• wherein at least one processing node of the plurality of processing nodes is to use the multidimensional data structure to process a query received through the control interface and to provide a query response, wherein at least one processing node of the plurality of processing nodes is to use modeled information to provide a suggested version of the plurality of selectable data processing parameters, and wherein at least one processing node of the plurality of processing nodes is to determine a plurality of sets of unique and persistent personal identifiers from the multidimensional data structure and to perform a plurality of set operations on the plurality of sets of unique and persistent personal identifiers.