Systems and methods for correlating large datasets of electronic data records
AssigneeNasdaq Technology AB
InventorCorentin François Arthur Plouet
Life of this patent
Estimating the term…
Abstract
A computer system is provided that stores reference data records and event data records that each correspond to one of the reference data records. Tokens are generated from the reference data records and sorted into subsets. Each reference data record that has a given token is included in the corresponding subset. A correlation value is calculated between members (e.g., the reference data records) of the subsets. The correlation values are calculated by determining a common time, calculating a reference value, calculating a percentage difference from the reference value, and then determining a correlation value (e.g., a correlation coefficient) using the percentage difference.
Claims20
A data surveillance computer system for detecting manipulation in how transactions are processed across multiple disparate data transaction systems, the data surveillance computer system comprising:
- at least one hardware processor that is configured to execute computer executable instructions to perform operations comprising:
- processing transaction records that each correspond to a different data transaction that was processed by a corresponding transaction processing computer system of a plurality of transaction processing computer systems, each of the transaction records including a first value of a corresponding data transaction and time data for when the corresponding data transaction was processed by the corresponding transaction processing computer system, each of the transaction records being linked to a corresponding entity record of a plurality of entity records;
- for each respective pair between members of a first plurality of paired entity records that are determined from the plurality of entity records, calculating a correlation value for the respective pair, wherein one of the respective pair(s) includes a first pair;
- in connection with the calculating the correlation value for the first pair that is between a first entity record and a second entity record:
- (a) calculating at least one common value that is based on at least one of the transaction records that are linked with the first entity record and/or the second entity record, wherein at least some of the transaction records include transaction records associated with data transactions processed by different ones of the plurality of transaction processing computer systems, and
- (b) for each respective transaction record of a plurality of transaction records that are linked to the first entity record and/or the second entity record, calculating a difference based on the at least one common value and the first value of the respective transaction record, wherein the correlation value for the first pair is based on the calculated difference;
- and
- storing, to data storage, the correlation value that is calculated for each pair of entity records that are determined from the plurality of entity records.
A method performed a data surveillance computer system for detecting manipulation in transactions that are processed by disparate data transaction systems, the method comprising:
- processing transaction records that each correspond to a different data transaction that was processed by a corresponding transaction processing computer system of a plurality of transaction processing computer systems, each of the transaction records including a first value of a corresponding data transaction and time data for when the corresponding data transaction was processed by the corresponding transaction processing computer system, each of the transaction records being linked to a corresponding entity record of a plurality of entity records;
- for each respective pair between members of a first plurality of paired entity records that are determined from the plurality of entity records, calculating a correlation value for the respective pair, wherein one of the respective pair(s) includes a first pair;
- in connection with the calculating the correlation value for the first pair that is between a first entity record and a second entity record:
- (a) calculating at least one common value that is based on at least one of the transaction records that are linked with the first entity record and/or the second entity record, wherein at least some of the transaction records include transaction records associated with data transactions processed by different ones of the plurality of transaction processing computer systems, and
- (b) for each respective transaction record of a plurality of transaction records that are linked to the first entity record and/or the second entity record, calculating a difference based on the at least one common value and the first value of the respective transaction record, wherein the correlation value for the first pair is based on the calculated difference;
- and
- storing, to data storage, the correlation value that is calculated for each pair of entity records that are determined from the plurality of entity records.
A non-transitory computer readable storage medium having stored thereon computer readable instructions for use with a data surveillance computer system for detecting manipulation in transactions that are processed by disparate data transaction systems, the stored computer readable instructions comprising instructions that cause the computer system to perform operations comprising:
- processing transaction records that each correspond to a different data transaction that was processed by a corresponding transaction processing computer system of a plurality of transaction processing computer systems, each of the transaction records including a first value of a corresponding data transaction and time data for when the corresponding data transaction was processed by the corresponding transaction processing computer system, each of the transaction records being linked to a corresponding entity record of a plurality of entity records;
- for each respective pair between members of a first plurality of entity records that are determined to be paired from the plurality of entity records, calculating a correlation value for the respective pair, wherein one of the respective pair(s) includes a first pair;
- in connection with the calculating the correlation value for the first pair that is between a first entity record and a second entity record:
- (a) calculating at least one common value that is based on at least one of the transaction records that are linked with the first entity record and/or the second entity record, wherein at least some of the transaction records include transaction records associated with data transactions processed by different ones of the plurality of transaction processing computer systems, and
- (b) for each respective transaction record of a plurality of transaction records that are linked to the first entity record and/or the second entity record, calculating a difference based on the at least one common value and the first value of the respective transaction record, wherein the correlation value for the first pair is based on the calculated difference;
- and
- storing, to data storage, the correlation value that is calculated for each pair of entity records that are determined from the plurality of entity records.
Dependencies are inferred from the claim text.