Data item classification and organization in large data sets

A data intake device for storing large data sets having a plurality of data entries each including at least one data item for classification, comprising:

• a network interface to a data set for classification;
• a log for capturing a set of raw rules resulting from inputs indicative of one or more evaluations and a conclusion of data classification operations, the data classification operations resulting in the conclusion for determining successive privacy handling;
• a rule analysis process for identifying at least one pattern in the set of raw rules by consolidating duplicative conditions and eliminating inconsequential conditions;
• and
• a rule table for applying the identified patterns to other data entries from the large data repository for defining the conclusions directing successive privacy handling of the data items in other data entries in the large data repository.

A computer program embodying program code on a non-transitory computer readable medium that, when executed by a processor, performs steps for implementing a method of classifying log messages into groups indicative of a message format, the method comprising:

• capturing a set of raw rules resulting from inputs indicative of evaluations and conclusions of data classification operations;
• identifying patterns in the set of raw rules by consolidating duplicative conditions and eliminating inconsequential conditions;
• applying a context to the rules based on a usage or domain of the data;
• and
• anonymizing the data and combining with other sets of anonymized data.