Multistage data sniffer for data extraction

A system for extracting data, the system comprising:

• a non-transitory memory having machine executable instructions;
• and
• a processing unit for accessing the memory and executing the machine executable instructions, the machine executable instructions comprising a multistage data sniffer that executes a plurality of multistage data sniffer instances in parallel, each multistage data sniffer instance comprising:
• a first stage that scans a given file extracted from a predetermined location for required data fields and optional data fields in a set of data fields based on a configuration file defining a selected format of the given file;
• a second stage that evaluates a value in each data filed in the set of data fields for the selected format to determine a validity of values in the set of data fields;
• and
• a third stage that extracts data within the plurality of fields of the given file, aggregates the data based on a predetermined set of rules defined in the configuration file and outputs a data to a data mart database characterizing the aggregated data.

A method for extracting data, the method comprising:

• executing, by a multistage data sniffer executing on a computing platform, a plurality of multistage data sniffer instances in parallel, wherein each multistage data sniffer instance executes a sub-method comprising:
• scanning a given file extracted from a predetermined location for required data fields and optional data fields in a set of data fields based on a configuration file defining a selected format of the given file;
• validating a value in each data filed in the set of data fields for the selected format to determine a validity of values in the set of data fields;
• extracting data within the plurality of fields of the given file;
• aggregating the data based on a predetermined set of rules defined in the configuration file;
• and
• outputting data to a data mart database characterizing the aggregated data.