Wordbreak algorithm with offset mapping

A computerized method, comprising:

• storing an original string formed of a plurality of characters;
• performing a wordbreak algorithm on the original string;
• tokenizing the original string to generate a processed string including a plurality of word tokens separated by one or more spaces;
• generating an offset map between locations within the word tokens in the processed string and corresponding locations in the original string, the offset map including a mapping between a first data structure including character offset index values in the original string and a second data structure including character offset index values in the processed string;
• classifying a portion of the processed string as a target by determining a start character offset index value and end character offset index value of the target in the processed string from among the character offset index values in the second data structure;
• identifying target characters in the original string that correspond to the target using the mapping between the character offset values in the first and second data structures in the offset map;
• and
• performing a predetermined action on the target characters in the original string.

A computer system configured to classify a word, comprising:

• a server computing device configured to execute a search program, the search program being configured to receive as input sensitive data definitions and one or more policies, and to search a data set including a plurality of original strings for sensitive data according to the sensitive data definitions, wherein the server computing device is configured to:
• perform a wordbreak algorithm on a selected original string from the plurality of original strings;
• tokenize the selected original string to generate a processed string including a plurality of word tokens separated by one or more spaces;
• generate an offset map between locations within word tokens in the processed string and corresponding locations in the original string, the offset map including a mapping between a first data structure including character offset index values in the original string and a second data structure including character offset index values in the processed string;
• classify a portion of the processed string as a target by determining a start character offset index value and end character offset index value of the target in the processed string from among the character offset index values in the second data structure;
• identify target characters in the original string that correspond to the target using the mapping between the character offset values in the first and second data structures in the offset map;
• and
• perform a predetermined action on the target characters in the original string.