Data content identification

A non-transitory computer-readable medium containing instructions to configure at least one processor to perform operations comprising:

• accessing a source file having one or more data records, the one or more data records associated with one or more data values arranged into at least one column having an unknown data type, the unknown data type including a mislabeled data type;
• and
• proposing one or more data types for the at least one column having the unknown data type by at least:
• examining the one or more data values in the at least one column, the examining including examining, based on pattern matching, the one or more data values in the at least one column and examining, based on data directory matching, the one or more data values with one or more entries in at least one data directory, wherein the examining the one or more data values in the at least one column is based on at least a match between a format of the one or more data values with one or more patterns and/or a match between the one or more data values with one or more entries in the at least one data directory,
• calculating, for the at least one column, one or more confidence scores as one or more percentages indicating, based on pattern matching and the data directory matching, how many of the one or more data values in the at least one column match and/or the at least one data directory match, wherein the one or more confidence scores are based on a first percentage of data values in the at least one column having a format that matches the one or more patterns and/or a second percentage of data values in the at least one column that match the one or more entries in the at least one data directory,
• in response to the one or more confidence scores being over a threshold score, assigning the one or more proposed data types for the at least one column, wherein the one or more confidence scores prioritize the one or more proposed data types for the at least one column,
• in response to the one or more confidence scores being below the threshold score, applying one or more context rules to the at least one column to determine, based on a neighboring data type contained in a neighboring column proximate to the at least one column, the one or more proposed data types, and
• in response to the one or more confidence scores being below the threshold score, assigning, based on the applying of the context rules, the determined one or more proposed data types for the at least one column, wherein the one or more proposed data types have been adjusted, due to the applied one or more context rules, one or more confidence scores and corresponding priorities,
• wherein the accessing and the proposing are performed by at least one processor.

A system comprising:

• at least one processor;
• and
• at least one memory, wherein the at least one processor and the at least one memory are configured to perform operations comprising:
• accessing a source file having one or more data records, the one or more data records associated with one or more data values arranged into at least one column having an unknown data type, the unknown data type including a mislabeled data type;
• and
• proposing one or more data types for the at least one column having the unknown data type, the proposed one or more data types determined via a two-stage analysis comprising:
• a first-stage analysis comprising:
• examining the one or more data values in the at least one column, the examining including examining, based on pattern matching, the one or more data values in the at least one column and examining, based on data directory matching, the one or more data values with one or more entries in at least one data directory, wherein the examining the one or more data values in the at least one column is based on at least a match between a format of the one or more data values with one or more patterns and/or a match between the one or more data values with one or more entries in the at least one data directory,
• calculating, for the at least one column, one or more confidence scores as one or more percentages indicating, based on pattern matching and the data directory matching, how many of the one or more data values in the at least one column match and/or the at least one data directory match, wherein the one or more confidence scores are based on a first percentage of data values in the at least one column having a format that matches the one or more patterns and/or a second percentage of data values in the at least one column that match the one or more entries in the at least one data directory, and
• in response to the one or more confidence scores being over a threshold score, assigning the one or more proposed data types for the at least one column, wherein the one or more confidence scores prioritize the one or more proposed data types for the at least one column, and
• a second-stage comprising:
• in response to the one or more confidence scores being below the threshold score, applying one or more context rules to the at least one column to determine, based on a neighboring data type contained in a neighboring column proximate to the at least one column, the one or more proposed data types, and
• in response to the one or more confidence scores being below the threshold score, assigning, based on the applying of the context rules, the determined one or more proposed data types for the at least one column, wherein the one or more proposed data types have been adjusted, due to the applied one or more context rules, one or more confidence scores and corresponding priorities.