Parallel unicode tokenization in a distributed network environment

A non-transitory computer-readable storage medium storing executable instructions that, when executed by a hardware processor, cause the hardware processor to perform steps comprising:

• receiving, at a local computing system, a string of characters in a character domain;
• accessing, by the local computing system, a set of token tables, each token table mapping, for each character in the character domain, a code value corresponding to the character to a different token value;
• instantiating, by the local computing system, a set of tokenization pipelines configured to operate in parallel to tokenize the string of characters, each tokenization pipeline corresponding to a different subset of the string of characters and configured to perform one or more sequential tokenization operations on the subset of the string of characters using a subset of the set of token tables to produce a tokenized set of characters, wherein a first sequential tokenization operation of a first tokenization pipeline operates on an output of a second sequential tokenization operation of a second tokenization pipeline, and wherein a third sequential tokenization operation of the second tokenization pipeline operates on an output of a fourth sequential tokenization operation of a third tokenization pipeline;
• and
• combining the tokenized sets of characters output by each tokenization pipeline and providing, by the local computing system, the combined tokenized sets of characters to a remote computing system.

A system comprising a hardware processor and a non-transitory computer-readable storage medium storing executable instructions that, when executed by the hardware processor, cause the hardware processor to perform steps comprising:

• receiving, at a local computing system, a string of characters in a character domain;
• accessing, by the local computing system, a set of token tables, each token table mapping, for each character in the character domain, a code value corresponding to the character to a different token value;
• instantiating, by the local computing system, a set of tokenization pipelines configured to operate in parallel to tokenize the string of characters, each tokenization pipeline corresponding to a different subset of the string of characters and configured to perform one or more sequential tokenization operations on the subset of the string of characters using a subset of the set of token tables to produce a tokenized set of characters, wherein a first sequential tokenization operation of a first tokenization pipeline operates on an output of a second sequential tokenization operation of a second tokenization pipeline, and wherein a third sequential tokenization operation of the second tokenization pipeline operates on an output of a fourth sequential tokenization operation of a third tokenization pipeline;
• and
• combining the tokenized sets of characters output by each tokenization pipeline and providing, by the local computing system, the combined tokenized sets of characters to a remote computing system.