Method and server for processing text sequence for machine processing task

A computer-implemented method for preparing a text sequence for a machine processing task to be performed by a Machine Learning Algorithm (MLA), the method executable by a server, the method comprising:

• acquiring, by the server, a token vocabulary, the token vocabulary storing a set of tokens from a pre-determined corpus of text, a given token from the set of tokens being one of a single character and a merged set of tokens;
• acquiring, by the server, a merge table, the merge table indicating possible mergers between pairs of tokens from the set of tokens, a given token from a given possible merge being associated with a respective frequency of occurrence of the given token in the pre-determined corpus of text;
• acquiring, by the server, the text sequence indicative of at least one word;
• for a given word from the text sequence:
• using, by the server, the token vocabulary for splitting the given word into an initial token sequence, the initial token sequence representing individual characters of the given word;
• iteratively merging, by the server, tokens of the initial token sequence to generate a final token sequence for the given word, the iteratively merging including:
• at a given merging iteration:
• using, by the server, the merge table for identifying a set of possible merges between pairs of adjacent tokens in a current token sequence of the given merging iteration;
• using, by the server, a pre-determined probability for selectively including possible merges from the set of possible merges into an augmented set of possible merges for the given merging iteration;
• using, by the server, the augmented set of possible merges for generating a new token sequence by performing at least one of the augmented set of possible merges in the current token sequence,
• the new token sequence to be used by the server as a current token sequence during a next merging iteration,
• at an other given merging iteration after the given merging iteration:
• in response to no more merges being available for a current token sequence of the other given merging iteration,
• identifying, by the server, the current token sequence of the other given merging iteration as the final token sequence to be used for the machine processing task.

A server for processing a text sequence for a machine processing task to be performed by a Machine Learning Algorithm (MLA), the server having access to the MLA, the server being configured to:

• acquire a token vocabulary, the token vocabulary storing a set of tokens from a pre-determined corpus of text, a given token from the set of tokens being one of a single character and a merged set of tokens;
• acquire a merge table, the merge table indicating possible mergers between pairs of tokens from the set of tokens, a given token from a given possible merge being associated with a respective frequency of occurrence of the given token in the pre-determined corpus of text;
• acquire the text sequence indicative of at least one word;
• for a given word from the text sequence:
• use the token vocabulary for splitting the given word into an initial token sequence, the initial token sequence representing individual characters of the given word;
• iteratively merge tokens of the initial token sequence to generate a final token sequence for the given word, the iteratively merging including:
• at a given merging iteration:
• use the merge table for identifying a set of possible merges between pairs of adjacent tokens in a current token sequence of the given merging iteration;
• exclude at least one of the set of possible merges based on a pre-determined dropout probability,
• thereby generating a reduced set of possible merges for the given merging iteration,
• the reduced set of possible merges being smaller than the set of possible merges;
• use the reduced set of possible merges for generating a new token sequence by performing at least one of the reduced set of possible merges in the current token sequence,
• the new token sequence to be used by the server as a current token sequence during a next merging iteration,
• at an other given merging iteration after the given merging iteration:
• in response to no more merges being available for a current token sequence of the other given merging iteration,
• identify the current token sequence of the other given merging iteration as the final token sequence to be used for the machine processing task.

A computer-implemented method for processing a text sequence for a machine processing task to be performed by a Machine Learning Algorithm (MLA), the method executable by a server, the method comprising:

• acquiring, by the server, a token vocabulary, the token vocabulary storing a set of tokens from a pre-determined corpus of text, a given token from the set of tokens being one of a single character and a merged set of tokens;
• acquiring, by the server, a merge table, the merge table indicating possible mergers between pairs of tokens from the set of tokens, a given token from a given possible merge being associated with a respective frequency of occurrence of the given token in the pre-determined corpus of text;
• acquiring, by the server, the text sequence indicative of at least one word;
• for a given word from the text sequence:
• using, by the server, the token vocabulary for splitting the given word into an initial token sequence, the initial token sequence representing individual characters of the given word;
• iteratively merging, by the server, tokens of the initial token sequence to generate a final token sequence for the given word, the iteratively merging including:
• at a given merging iteration:
• using, by the server, the merge table for identifying a set of possible merges between pairs of adjacent tokens in a current token sequence of the given merging iteration;
• using, by the server, a stochastically-driven algorithm for excluding at least one of the set of possible merges,
• thereby generating a reduced set of possible merges for the given merging iteration,
• the reduced set of possible merges being smaller than the set of possible merges;
• using, by the server, the reduced set of possible merges for generating a new token sequence by performing at least one of the reduced set of possible merges in the current token sequence,
• the new token sequence to be used by the server as a current token sequence during a next merging iteration,
• at an other given merging iteration after the given merging iteration:
• in response to no more merges being available for a current token sequence of the other given merging iteration,
• identifying, by the server, the current token sequence of the other given merging iteration as the final token sequence to be used for the machine processing task.