Systems and methods for correlating large datasets of electronic data messages

A computer system comprising:

• an electronic storage system configured to store:
• a plurality of reference data records, and
• transaction request records regarding data transaction requests processed by other computer systems, each of the transaction request records having a numerical value and corresponding to at least one of the plurality of reference data records;
• a processing system that includes at least one hardware processor, the processing system configured to:
• generate a plurality of reference groups that each include multiple reference data records, and
• determine, for each of the plurality of reference groups, a correlation value between at least two reference data records within the same reference group, wherein determination of the correlation value is based on at least:
• a calculation of a common time window for which the transaction requests, which correspond to the at least two reference data records that are within the same reference group, were processed by the other computer systems,
• a calculation, which is based on the common time window, of a starting value for the at least two reference data records, and
• a calculation, for each of the at least two reference data records within the same reference group, of a percentage difference between the starting value and the numerical value of a transaction request record,
• wherein the correlation value between at least two reference data records within the same reference group is based on the percentage difference.

A method performed a computer system that includes an electronic storage system and a processing system that includes at least one hardware processor, the method comprising:

• storing, to the electronic storage system, a plurality of reference data records;
• storing, to the electronic storage system, event messages that each include data regarding processed data transaction requests, each of the event messages corresponding to one of the plurality of reference data records and having a numerical value;
• generating a plurality of subsets that each include at least two of the plurality of reference data records;
• performing a correlation process for a first subset of the plurality of subsets that at least includes (1)-(4):
• (1) calculating a common time window for which the transaction requests, which correspond to the at least two reference data records that are within the first subset reference group, were processed,
• (2) calculating, based on the common time window and the event messages, a starting value for the at least two reference data records that are within the first subset reference group;
• (3) calculating, for each of the at least two reference data records that are within the first subset reference group, a percentage difference between the starting value and the numerical value of at least one of the event messages;
• and
• (4) calculating a correlation value between the at least two reference data records based on the calculated percentage difference.

A non-transitory computer readable storage medium having stored thereon computer readable instructions for use with a computer system that includes at least one processor, electronic memory, the electronic memory configured to store (1) a plurality of reference data records, and (2) event messages that each include data regarding data transaction requests processed by other computer systems, each of the event messages corresponding to one of the plurality of reference data records and having a numerical value field, the stored computer readable instructions comprising instructions that cause the computer system to:

• select a first reference data record and a second reference data record, both being included in the plurality of reference data records;
• determine, using event messages that correspond to the first and second reference data records, a common time window for which the corresponding event messages were processed by the other computer systems;
• (a) calculate, based on the common time window and the corresponding event messages, a starting value both first and second reference data records;
• (b) calculate a first percentage difference between the starting value and a numerical value from of at least one event message that corresponds to the first reference data record;
• (c) calculate a second percentage difference between the starting value and a numerical value from of at least one event message that corresponds to the second reference data record;
• (d) calculate and then store a correlation value between the first reference data record and the second reference data record using the calculated first and second percentage difference.
• repeat (a)-(d) for different pairs of reference data records of the plurality of reference data records.