Distributed processing of data records

A computer program product for distributed processing of data on a distributed database computer system, the computer program product comprising a non-transitory computer readable storage medium having program instructions embodied therewith, the program instructions executable by a processor to cause the processor to:

• distribute rows of an arbitrary matrix to all of a plurality of processing units, wherein a number of the rows is equal to a number of the processing units, wherein a matrix set is used to calculate a value set corresponding to each data record by using said data record as input, the matrix set comprising at least one processing parameter matrix, the value set comprising at least one calculation value, the distributed database computer system comprising the plurality of processing units connected in a share-nothing parallel processing architecture, wherein each processing unit comprises a processor of the each processing unit, a processor memory of the each processing unit, and a storage memory of the each processing unit, wherein the arbitrary matrix is stored in the distributed database in a way that one row of the arbitrary matrix is stored in each storage memory, wherein the data records and the matrix set are stored in a distributed database using the storage memories, each processor is operable for executing user defined functions (UDFs), calculating the value set corresponding to only one data record at a time, executing transaction processing, storing data in the processor memory, and using the data stored in the processor memory for execution of the UDFs within a framework of one transaction;
• and perform transaction processing in a framework of one transaction by:
• executing a first UDF on each processing unit, wherein a Cartesian product of the each processing parameter matrix and the row of the arbitrary matrix is calculated on each processing unit and as a result thereof the matrix set is stored in the processor memory of each processing unit;
• and
• executing a second UDF on each processing unit having at least one data record after the executing of the first UDF, wherein a number of repetitive executions of the second UDF on each processing unit is equal to the number of the data records stored in the storage memory of the each processing unit and all data records stored in the storage memory of the each processing unit are processed one by one, wherein the value set corresponding to the data record is calculated using the matrix set stored in the processor memory of said respective processing unit, wherein the matrix set is used to calculate the value set corresponding to each data record by using said data record as input of a link function of a generalized linear model;
• wherein the at least one processing parameter matrix of the matrix set includes at least the following set of processing parameter matrices:
• a predictor-to-parameter correlation matrix and a beta coefficient matrix, wherein the predictor-to-parameter correlation matrix determines a correlation between each calculation value of the value set and a corresponding data record in said link function, wherein the predictor-to-parameter correlation matrix comprises rows of the predictor-to-parameter correlation matrix and each row of the predictor-to-parameter correlation matrix is used in conjunction with a corresponding beta coefficient of the beta coefficient matrix in said link function.

A computer system for distributed processing of data on a distributed database computer system, the computer system comprising:

• a memory;
• and
• a processor, communicatively coupled to said memory, the computer system configured to perform a method comprising:
• distributing rows of an arbitrary matrix to all of a plurality of processing units, wherein a number of the rows is equal to a number of the processing units, wherein a matrix set is used to calculate a value set corresponding to each data record by using said data record as input, the matrix set comprising at least one processing parameter matrix, the value set comprising at least one calculation value, the distributed database computer system comprising the plurality of processing units connected in a share-nothing parallel processing architecture, wherein each processing unit comprises a processor of the each processing unit, a processor memory of the each processing unit, and a storage memory of the each processing unit, wherein the arbitrary matrix is stored in the distributed database in a way that one row of the arbitrary matrix is stored in each storage memory, wherein the data records and the matrix set are stored in a distributed database using the storage memories, each processor is operable for executing user defined functions (UDFs), calculating the value set corresponding to only one data record at a time, executing transaction processing, storing data in the processor memory, and using the data stored in the processor memory for execution of the UDFs within a framework of one transaction;
• and
• performing transaction processing in a framework of one transaction by:
• executing a first UDF on each processing unit, wherein a Cartesian product of the each processing parameter matrix and the row of the arbitrary matrix is calculated on each processing unit and as a result thereof the matrix set is stored in the processor memory of each processing unit;
• and
• executing a second UDF on each processing unit having at least one data record after the executing of the first UDF, wherein a number of repetitive executions of the second UDF on each processing unit is equal to the number of the data records stored in the storage memory of the each processing unit and all data records stored in the storage memory of the each processing unit are processed one by one, wherein the value set corresponding to the data record is calculated using the matrix set stored in the processor memory of said respective processing unit, wherein the matrix set is used to calculate the value set corresponding to each data record by using said data record as input of a link function of a generalized linear model;
• wherein the at least one processing parameter matrix of the matrix set includes at least the following set of processing parameter matrices:
• a predictor-to-parameter correlation matrix and a beta coefficient matrix, wherein the predictor-to-parameter correlation matrix determines a correlation between each calculation value of the value set and a corresponding data record in said link function, wherein the predictor-to-parameter correlation matrix comprises rows of the predictor-to-parameter correlation matrix and each row of the predictor-to-parameter correlation matrix is used in conjunction with a corresponding beta coefficient of the beta coefficient matrix in said link function.