Use of small unit language model for training large unit language models

A computer-implemented method, comprising:

• generating a plurality of sequences of small unit tokens from a first language model that is trained with a small unit corpus including the small unit tokens, the small unit corpus having been derived by tokenization with a small unit;
• tokenizing the plurality of sequences of small unit tokens by a large unit that is larger than the small unit, to create a derived large unit corpus including derived large unit tokens, the large unit corpus being an in-domain corpus;
• re-tokenizing the derived large unit corpus by the small unit to re-create the small unit corpus including the small unit tokens, the large unit corpus including the derived large unit tokens and having been derived by tokenization with the large unit;
• and
• dynamically training a plurality of word language models using combined word sequences associated with an out-of-domain native text corpus and the in-domain corpus, the large unit tokens being utilized in the out-of-domain corpus during the training and being derived by joining a randomly determined number of sequential small unit tokens in the plurality of sequences of small unit tokens, the plurality of word language models trained using the small unit corpus and the large unit corpus being interpolated to generate an interpolated language model.

An apparatus comprising:

• a processor or a programmable circuitry;
• and
• one or more computer readable mediums collectively including instructions that, when executed by the processor or the programmable circuitry, cause the processor or the programmable circuitry to perform operations comprising:
• generating a plurality of sequences of small unit tokens from a first language model that is trained with a small unit corpus including the small unit tokens, the small unit corpus having been derived by tokenization with a small unit;
• tokenizing the plurality of sequences of small unit tokens by a large unit that is larger than the small unit, to create a derived large unit corpus including derived large unit tokens, the large unit corpus being an in-domain corpus;
• re-tokenizing the derived large unit corpus by the small unit to re-create the small unit corpus including the small unit tokens, the large unit corpus including the derived large unit tokens and having been derived by tokenization with the large unit;
• and
• dynamically training a plurality of word language models using combined word sequences associated with an out-of-domain native text corpus and the in-domain corpus, the large unit tokens being utilized in the out-of-domain corpus during the training and being derived by joining a randomly determined number of sequential small unit tokens in the plurality of sequences of small unit tokens, the plurality of word language models trained using the small unit corpus and the large unit corpus being interpolated to generate an interpolated language model.