Domain-specific language models

A computer-implemented method comprising:

• receiving domain-specific text data;
• receiving a language model;
• training the language model using the domain-specific text data, thereby generating a pretrained domain-specific language model;
• modifying the domain-specific text data using the pretrained domain-specific language model, thereby creating word embeddings;
• applying global average pooling to the word embeddings by applying average pooling to the word embeddings for all tokens and submitting the word embeddings to a plurality of different architectures including one or more dense layers with dropout, and one or more dense layers without dropout;
• feeding the word embeddings into one or more dense layers of a neural network, thereby training a classification task;
• and
• performing the classification task using the pretrained domain-specific language model.

A system comprising:

• a first one or more electronic devices to implement a model hosting system in a multi-tenant provider network;
• and
• a second one or more electronic devices to implement a model training system in the multi-tenant provider network, the model training system including instructions that, upon execution by one or more processors, cause the model training system to:
• receive domain-specific text data;
• receive a language model;
• train the language model using the domain-specific text data, thereby generating a pretrained domain-specific language model;
• modify the domain-specific text data using the pretrained domain-specific language model, thereby creating word embeddings;
• apply global average pooling to the word embeddings by applying average pooling to the word embeddings for all tokens and submitting the word embeddings to a plurality of different architectures including one or more dense layers with dropout, and one or more dense layers without dropout;
• feed the word embeddings into one or more dense layers of a neural network, thereby training a classification task;
• and
• perform the classification task using the pretrained domain-specific language model.