Cross-lingual regularization for multilingual generalization

A system for training a natural language processing (NLP) deep learning module, the system comprising:

• a memory;
• and
• one or more processors;
• wherein the one or more processors are configured to:
• obtain a first dataset of training data in a first language;
• perform one or more training passes of:
• adding a new language to a set of regularization languages different from the first language,
• creating a second dataset of training data translated from the first dataset of training data corresponding to the set of regularization languages,
• training the deep learning module using the second dataset, and
• determining whether a performance improvement metric associated with the trained deep learning module is greater than a threshold;
• and
• select a trained deep learning module with a best performance improvement metric over the one or more training passes.

A non-transitory machine-readable medium comprising executable code which when executed by one or more processors associated with a computing device are adapted to cause the one or more processors to perform a method comprising:

• obtaining a first dataset of training data in a first language;
• performing one or more training passes of:
• adding a new language to a set of regularization languages different from the first language,
• creating a second dataset of training data translated from the first dataset of training data corresponding to the set of regularization languages,
• training the deep learning module using the second dataset, and
• determining whether a performance improvement metric associated with the trained deep learning module is greater than a threshold;
• and
• selecting a trained deep learning module with a best performance improvement metric over the one or more training passes.