Method/system for the online identification and blocking of privacy vulnerabilities in data streams

A system comprising:

• at least one data processor connected to at least one memory that stores software instructions, where execution of the software instructions by the at least one data processor causes the system to automatically identify and protect privacy vulnerabilities in a plurality of data streams of data records received by a plurality of virtual machines by:
• receiving at least one data stream at each local virtual machine, the data stream including data values for a plurality of attributes;
• indexing the data values for each attribute of the at least one data stream received by each local virtual machine based on a schema of each data stream;
• integrating by a main virtual machine the local virtual machine indexes into a global index data structure for the plurality of data streams, the global index data structure including single attribute data values;
• identifying privacy vulnerabilities in the data as:
• attributes that are direct identifiers based on the attribute data values stored in the global index data structure;
• combinations of attributes that are quasi-identifiers based on the low frequency of certain combinations of attribute data value pairs by computing the frequency based on the single attribute data values stored in the global index data structure;
• and
• providing privacy protection to the data streams by applying data transformations on the discovered direct identifiers and the quasi-identifiers.

A computer program product comprising:

• software instructions on a non-transitory computer-readable medium, where execution of the software instructions using a computer causes the computer to automatically identify and protect privacy vulnerabilities in a plurality of data streams of data records received by a plurality of virtual machines by:
• receiving at least one data stream at each local virtual machine, the data stream including data values for a plurality of attributes;
• indexing the data values for each attribute of the at least one data stream received by each local virtual machine based on a schema of each data stream;
• integrating by a main virtual machine the local virtual machine indexes into a global index data structure for the plurality of data streams, the global index data structure including single attribute data values;
• identifying privacy vulnerabilities in the data as:
• attributes that are direct identifiers based on the attribute data values stored in the global index data structure;
• combinations of attributes that are quasi-identifiers based on the low frequency of certain combinations of attribute data value pairs by computing the frequency based on the single attribute data values stored in the global index data structure;
• providing privacy protection to the data streams by applying data transformations on the discovered direct identifiers and the quasi-identifiers;
• and
• classifying the data attributes of the plurality of data streams into known data types, the classification being performed either before identifying the privacy vulnerabilities in the data, or after identifying the privacy vulnerabilities in the data.