US 11,537,950 B2 · Utilizing a joint-learning self-distillation framework for improving text sequential labeling machine-learning models

Abstract

This disclosure describes one or more implementations of a text sequence labeling system that accurately and efficiently utilize a joint-learning self-distillation approach to improve text sequence labeling machine-learning models. For example, in various implementations, the text sequence labeling system trains a text sequence labeling machine-learning teacher model to generate text sequence labels. The text sequence labeling system then creates and trains a text sequence labeling machine-learning student model utilizing the training and the output of the teacher model. Upon the student model achieving improved results over the teacher model, the text sequence labeling system re-initializes the teacher model with the learned model parameters of the student model and repeats the above joint-learning self-distillation framework. The text sequence labeling system then utilizes a trained text sequence labeling model to generate text sequence labels from input documents.

Claims20

Dependent claims

A non-transitory computer-readable medium storing instructions that, when executed by at least one processor, cause a computer system to:

utilize a first set of learned model parameters of a first text sequence labeling machine-learning model to initialize a second text sequence labeling machine-learning model;
generate a set of ground truth text sequence labels for a text data set utilizing the first set of learned model parameters of the first text sequence labeling machine-learning model;
learn a second set of model parameters of the second text sequence labeling machine-learning model utilizing the text data set and the set of ground truth text sequence labels;
and
based on identifying an improved validation output from the second text sequence labeling machine-learning model, re-initialize the first text sequence labeling machine-learning model with the second set of learned model parameters.

2 more elements

A system comprising:

a memory device comprising a first text data set having a first set of ground truth text sequence labels and a second text data set;
and
at least one computing device configured to cause the system to:
learn a first set of model parameters of a first text sequence labeling machine-learning model utilizing the first text data set and the first set of ground truth text sequence labels;
initialize a second text sequence labeling machine-learning model utilizing the first set of learned model parameters;
determine a second set of ground truth text sequence labels for the second text data set utilizing the first text sequence labeling machine-learning model, the first set of learned model parameters, and the second text data set;
learn a second set of model parameters of the second text sequence labeling machine-learning model utilizing the first text data set, the first set of ground truth text sequence labels, the second text data set, and the second set of ground truth text sequence labels;
and
based on identifying an improved validation output from the second text sequence labeling machine-learning model, re-initialize the first text sequence labeling machine-learning model with the second set of learned model parameters.

6 more elements

In a digital medium environment for utilizing machine-learning models to generate text sequence labels from unlabeled documents, a computer-implemented method comprising:

utilizing a first set of learned model parameters of a first text sequence labeling machine-learning model to initialize a second text sequence labeling machine-learning model;
learning a second set of model parameters of the second text sequence labeling machine-learning model utilizing a text data set and a set of ground truth text sequence labels generated utilizing the first set of learned model parameters of the first text sequence labeling machine-learning model;
based on identifying an improved validation output from the second text sequence labeling machine-learning model having the second set of learned model parameters, re-initializing the first text sequence labeling machine-learning model with the second set of learned model parameters;
and
learning a third set of model parameters of the second text sequence labeling machine learning model utilizing one or more text data sets and an additional set of ground truth text sequence labels generated utilizing the second set of learned model parameters of the first text sequence labeling machine-learning model.

2 more elements

Dependencies are inferred from the claim text.

Utilizing a joint-learning self-distillation framework for improving text sequential labeling machine-learning models

Life of this patent

Utilizing a joint-learning self-distillation framework for improving text sequential labeling machine-learning models

Life of this patent

Abstract

Claims20