Techniques for Pretraining Document Language Models for Example-Based Document Classification

A method implemented in a data processing system for classifying a document, the method comprising:

• receiving a set of one or more unlabeled documents associated one or more first categories of documents to be used to train machine learning models to analyze the one or more unlabeled documents;
• and
• fine-tuning a first machine learning model and a second machine learning model based on the one or more unlabeled document to enable the first machine learning model to determine a semantic representation of the one or more first categories of document, and to enable the second machine learning model to classify the semantic representations according to the one or more first categories of documents, the first machine learning model and the second machine learning model having been trained using first unlabeled training data including a second plurality of categories of documents, the second plurality of categories of documents not including the one or more first categories of documents.

A machine-readable medium on which are stored instructions that, when executed, cause a processor of a programmable device to perform operations of:

• receiving a set of one or more unlabeled documents associated one or more first categories of documents to be used to train machine learning models to analyze the one or more unlabeled documents;
• and
• fine-tuning a first machine learning model and a second machine learning model based on the one or more unlabeled document to enable the first machine learning model to determine a semantic representation of the one or more first categories of document, and to enable the second machine learning model to classify the semantic representations according to the one or more first categories of documents, the first machine learning model and the second machine learning model having been trained using first unlabeled training data including a second plurality of categories of documents, the second plurality of categories of documents not including the one or more first categories of documents.