Random token segmentation for training next token prediction models

A method, comprising:

• receiving, at a machine learning system, text data for training a machine learning model to predict a next text token given input text tokens;
• determining, by the machine learning system, a plurality of tokens from the text data;
• randomly, by the machine learning system, segmenting different ones of the plurality of tokens into respective pluralities of sub-tokens;
• and
• training, by the machine learning system, the machine learning model to predict the next text token given the input text tokens using the plurality of tokens including the respective pluralities of sub-tokens as a training data set.

One or more non-transitory, computer-readable storage media, storing program instructions that when executed on or across one or more computing devices cause the one or more computing devices to implement:

• receiving a text data for training a machine learning model to predict a next text token given input text tokens;
• determining a plurality of tokens from the text data;
• randomly segmenting different ones of the plurality of tokens into respective pluralities of sub-tokens;
• and
• causing training of the machine learning model to predict the next text token given the input text tokens using the plurality of tokens including the respective pluralities of sub-tokens as a training data set.