Data leakage and misuse detection

A data detection system, comprising:

• a hardware processor configured to:
• receive an evaluation dataset A;
• build a signature of the evaluation dataset A (sig(A)), wherein the signature of a dataset is a multi-level evaluation data abstraction representation of the dataset;
• build a signature for each of existing datasets B (B1, B2, . . . , Bn) (sig(Bx)) that are stored in a memory;
• compare the sig(A) with each of the sig(Bx)s;
• derive a similarity score based on the comparing;
• and
• responsive to determining the similarity score exceeds a predefined threshold, generate an activity related to the determination;
• wherein the derivation of the similarity score comprises the processor being configured to:
• create levels of respective signatures for each of the datasets by creating a single block at a highest level that represents the entire dataset and splitting each block into a plurality of blocks at a lower level that represent portions of the dataset;
• determine one or more blocks at each level for the sig(a) and the sig(Bx)s and calculating a digest value for each of the determined blocks;
• for each of the sig(Bx)s, perform a block comparing of the digest value for each block to a digest value in a corresponding block of the sig(a) to determine when the digest values match;
• and
• compute the similarity score based on a number of the block compares that match.

A method for identifying copied data comprising using a processor for:

• receiving an evaluation dataset A;
• building a signature of the evaluation dataset A (sig(A)) , wherein the signature of a dataset is a multi-level evaluation data abstraction representation of the dataset;
• building a signature for each of existing datasets B (B1, B2, . . . , Bn) (sig(Bx)) that are stored in a memory;
• comparing the sig(A) with each of the sig(Bx)s;
• deriving a similarity score based on the comparing;
• and
• responsive to determining the similarity score exceeds a predefined threshold, generating an activity related to the determination;
• wherein:
• the deriving of the similarity score comprises:
• creating levels of respective signatures for each of the datasets by creating a single block at a highest level that represents the entire dataset and splitting each block into a plurality of blocks at a lower level that represent portions of the dataset;
• determining one or more blocks at each level for the sig(a) and the sig(Bx)s and calculating a digest value for each of the determined blocks;
• for each of the sig(Bx)s, performing a block comparing of the digest value for each block to a digest value in a corresponding block of the sig(a) to determine when the digest values match;
• and
• computing the similarity score based on a number of the block compares that match;
• and
• no splitting of a particular block takes place conditioned upon the digest of the particular block being equal to a corresponding block in a dataset under comparison.