Systems and methods for tokenizing user-annotated names

A system for tokenizing user-annotated names, the system comprising:

• an identification module, stored in memory, that identifies an example set of user-annotated names that represent strings of text used to identify digital entities within a particular organization;
• a dictionary creation module, stored in memory, that creates, based on the example set of user-annotated names, a custom dictionary of known keywords by:
• extracting a set of known keywords from the example set of user-annotated names;
• assigning, based on the respective frequency of each of the known keywords within the example set of user-annotated names, a frequency score to each known keyword in the set of known keywords that represents a frequency at which the known keyword appears within the example set of user-annotated names;
• a tokenizing module, stored in memory, that enables a computing device to tokenize an additional user-annotated name of arbitrary structure by performing a semantic analysis comprising:
• applying a tokenization algorithm that, at each iterative step of the tokenization algorithm:
• accepts a text string as a string input;
• generates each possible single split of the string input, generating a total number of permutations equal to the length of the string input;
• uses, as an input to a current iterative step in the tokenization algorithm, a previous output of the tokenization algorithm that used one side of the single split as a previous string input to the tokenization algorithm;
• and
• assigns a frequency score to the other side of the single split based on whether the substring matches a known keyword stored in the custom dictionary;
• splitting the additional user-annotated name into tokens according to an output of the tokenization algorithm that used the additional user-annotated name as an initial input and received a top combined frequency score, thereby enabling a computing system to process the additional user-annotated name according to the tokens produced as the output of the tokenization algorithm;
• at least one physical processor configured to execute the identification module, the dictionary creation module, and the tokenizing module.

A non-transitory computer-readable medium comprising one or more computer-readable instructions that, when executed by at least one processor of a computing device, cause the computing device to:

• identify an example set of user-annotated names that represent strings of text used to identify digital entities within a particular organization;
• create, based on the example set of user-annotated names, a custom dictionary that comprises known keywords by:
• extracting a set of known keywords from the example set of user-annotated names;
• assigning, based on the respective frequency of each of the known keywords within the example set of user-annotated names, a frequency score to each known keyword in the set of known keywords that represents a frequency at which the known keyword appears within the example set of user-annotated names;
• enable tokenization of an additional user-annotated name of arbitrary structure by performing a semantic analysis comprising:
• applying a tokenization algorithm that, at each iterative step of the tokenization algorithm:
• accepts a text string as a string input;
• generates each possible single split of the string input, generating a total number of permutations equal to the length of the string input;
• uses, as an input to a current iterative step in the tokenization algorithm, a previous output of the tokenization algorithm that used one side of the single split as a previous string input to the tokenization algorithm;
• and
• assigns, using the custom dictionary, a frequency score to the other side of the single split based on whether the substring matches a known keyword stored in the custom dictionary;
• and
• splitting the additional user-annotated name into tokens according to an output of the tokenization algorithm that used the additional user-annotated name as an initial input and received a top combined frequency score, thereby enabling a computing system to process the additional user-annotated name according to the tokens produced as the output of the tokenization algorithm.