Salting text and fingerprinting in database tables, text files, and data feeds

A method of detecting the presence of one or more records from one of a plurality of data files in a wild file, wherein the plurality of data files and the wild file each comprise a plurality of records each comprising a plurality of fields, the method comprising the steps of:

• selected a plurality of sorting techniques from a set of sorting techniques in a sorting technique store;
• at a sorter engine, applying the plurality of sorting techniques from the sorting technique store to the wild file;
• for each application of one of the plurality of sorting techniques to the wild file, retrieving a plurality of salting techniques from a set of salting techniques in a salting techniques store, producing a plurality of salted data files using the plurality of salting techniques, wherein each of the plurality of available salting techniques comprises alteration of data in a record, and, for each of the plurality of salting techniques, comparing at a salting engine the results of salting at least one of the data files to the wild file to determine if a match occurs with one of the plurality of salted data files;
• at a fingerprint engine, producing a plurality of fingerprints, wherein each of the plurality of fingerprints is produced from at least one field of at least one record of each of the salted data files;
• at a match engine, comparing the results of salting the at least one data file to the wild file;
• if a match occurs after comparing the results of salting the at least one data file to the wild file, comparing at least one field of at least one record in the wild file to at least one of a plurality of fingerprints;
• if the comparison of the at least one field of at least one record in the wild file to at least one of the plurality of fingerprints results in a match, returning a result that the wild file was derived from the data file;
• and
• if for each application of one of the plurality of sorting techniques and each application of one of the plurality of salting techniques to the wild file does not result in a match with the fingerprint, returning a result that the wild file was not derived from the data file.

A system for salting and fingerprinting a data file, wherein the data file comprises a plurality of records, and each of such records comprises at least one field, comprising:

• a sorter engine configured to receive a data file, wherein the data file comprises a plurality of records each comprising at least one field, and wherein the sorter engine is configured to sort the records in the data file according to a sorting technique to produce a sorted data file, and to resort the sorted data file back to its original order to produce a resorted data file, and further wherein the sorter engine is configured to receive a wild file comprising a plurality of records each comprising at least one field;
• a sorting technique store in communication with the sorter engine, wherein the sorting technique store comprises a plurality of sorting techniques, and further wherein the sorter engine is further configured to apply each of the plurality of sorting techniques to the wild file;
• a salting engine in communication with the sorter engine and configured to receive the sorted data file and apply a salt to the sorted data file according to a salting technique to produce a salted data file;
• a salting technique store in communication with the salting engine, wherein the salting technique store comprises a plurality of salting techniques wherein each of the plurality of salting techniques comprises alteration of data in a record, and wherein the salting engine is further configured, for each application of one of the plurality of sorting techniques to the wild file, to retrieve each of the plurality of salting techniques from the salting technique store;
• a match engine configured to, for each of the plurality of salting techniques, comparing the results of salting the data file to the wild file to determine if a match occurs;
• a fingerprint engine in communication with the salting engine configured to produce a fingerprint from at least one field of at least one record of the salted data file and, if a match occurs after comparing the results of salting the data file to the wild file, compare at least one field of at least one record in the wild file to at least one of a plurality of fingerprints, and, if the comparison of the at least one field of at least one record in the wild file to at least one of the plurality of fingerprints results in a match, returning a result that the wild file was derived from the data file, or, if for each application of one of the plurality of sorting techniques and each application of one of the plurality of salting techniques to the wild file does not result in a match with at least one of the plurality of fingerprints, returning a result that the wild file was not derived from the data file;
• and
• a file database in communication with the sorter engine, the salting engine, and the fingerprint engine, wherein the file database is configured to store a data file identifier, a selected sorting technique identifier, a selected salting technique identifier, and the plurality of fingerprints.