Optimal Subword Tokenization and Vocabulary Creation

A system for subword tokenization, the system comprising:

• a storage device that stores program instructions;
• one or more processors operably connected to the storage device and configured to execute the program instructions to cause the system to:
• receive a text document comprising n bytes;
• specify a maximum token width of L bytes;
• define an initial vocabulary of tokens, wherein the tokens comprise a number of different n-grams of L or less bytes;
• tokenize the document with the fewest number of tokens from the vocabulary according to a minimum total weight through a directed acyclic graph comprising nodes that represent intervals between the bytes in the document and edges that represent potential tokens from the vocabulary appearing in text of the document;
• and
• perform natural language processing on the text document according to the tokenization.

A computer program product for subword tokenization, the computer program product comprising:

• a computer-readable storage medium having program instructions embodied thereon to perform the steps of:
• receiving a text document comprising n bytes;
• specifying a maximum token width of L bytes;
• defining an initial vocabulary of tokens, wherein the tokens comprise a number of different n-grams of L or less bytes;
• tokenizing the document with the fewest number of tokens from the vocabulary according to a minimum total weight through a directed acyclic graph comprising nodes that represent intervals between the bytes in the document and edges that represent potential tokens from the vocabulary appearing in text of the document;
• and
• performing natural language processing on the text document according to the tokenization.