Pretraining of split layer portions for multilingual model
AssigneeInternational Business Machines Corporation
InventorsLIN PAN, Haode Qi, Ladislav Kunc, Saloni Potdar
Life of this patent
Estimating the term…
Abstract
A method, computer system, and a computer program product for training a machine learning model are provided. A machine learning model may be split into a lower portion and an upper portion. The lower portion includes at least one layer. The upper portion includes at least one layer. The lower portion may be pre-trained via a generator task and via alternating between inputting of monolingual text data and multilingual text data. The upper portion may be pre-trained via a discriminator task. The pre-trained lower portion may be joined to the pre-trained upper portion to form a trained multilingual machine learning model.
Claims20
A method for training a machine learning model, the method comprising:
- splitting a machine learning model into a lower portion and an upper portion, the lower portion comprising at least one layer, the upper portion comprising at least one layer;
- pre-training the lower portion via a generator task and via alternating between inputting of monolingual text data and multilingual text data;
- pre-training the upper portion via a discriminator task;
- and
- joining the pre-trained lower portion to the pre-trained upper portion to form a trained multilingual machine learning model.
A computer system for training a multilingual machine learning model that performs natural language processing, the computer system comprising:
- one or more processors, one or more computer-readable memories, one or more computer-readable tangible storage media, and program instructions stored on at least one of the one or more computer-readable tangible storage media for execution by at least one of the one or more processors via at least one of the one or more computer-readable memories to cause the computer system to:
- split a machine learning model into a lower portion and an upper portion, the lower portion comprising at least one layer, the upper portion comprising at least one layer;
- pre-train the lower portion via a generator task and via alternating between inputting of monolingual text data and multilingual text data;
- pre-train the upper portion via a discriminator task;
- and
- join the pre-trained lower portion to the pre-trained upper portion to form a trained multilingual machine learning model.
A computer program product for training a multilingual machine learning model that performs natural language processing, the computer program product comprising a computer-readable storage medium having program instructions embodied therewith, wherein the program instructions are executable by a processor to cause the processor to perform a method comprising:
- split a machine learning model into a lower portion and an upper portion, the lower portion comprising at least one layer, the upper portion comprising at least one layer;
- pre-train the lower portion via a generator task and via alternating between inputting of monolingual text data and multilingual text data;
- pre-train the upper portion via a discriminator task;
- and
- join the pre-trained lower portion to the pre-trained upper portion to form a trained multilingual machine learning model.
Dependencies are inferred from the claim text.