US 12,614,066 B2Granted

Distilling transformers for neural cross-domain search

AssigneeMicrosoft Technology Licensing, LLC

InventorsColin Bruce Clement, Dawn Drain, Neelakantan Sundaresan, Chen Wu

Life of this patent

Estimating the term…

Abstract

A distillation system extracts knowledge from a large pre-trained sequence-to-sequence neural transformer model into a smaller bi-encoder. The pre-trained sequence-to-sequence neural transformer model is trained to translate data from a first domain into a second domain on a large corpus. A teacher model is generated from the pre-trained model by fine-tuning the pre-trained neural transformer model on a smaller translation task with true translation pairs. The fine-tuned model is then used to generate augmented data values which are used with the true translation pairs to train the bi-encoder. The bi-encoder is used for perform cross-domain searches.

Claims20

Dependent claims

A system for training a bi-encoder for a neural cross-domain search, comprising:

a processor;
and
a memory that stores a program configured to be executed by the processor, wherein the program comprises instructions to perform acts that:
obtain a pre-trained sequence-to-sequence deep learning model that produces a conditional-probability distribution to translate a sequence in a first domain into a sequence in a second domain and to translate the sequence in the second domain into the sequence in the first domain, wherein the first domain and the second domain differ, wherein the pre-trained sequence-to-sequence deep learning model comprises at least one encoder and at least one decoder, wherein the sequence in the first domain comprises natural language text and/or source code, wherein the sequence in the second domain comprises natural language text and/or source code;
obtain a plurality of true translation pairs, wherein a true translation pair of the plurality of true translation pairs comprises a first sequence in the first domain and a second sequence in the second domain;
train the pre-trained sequence-to-sequence deep learning model with the plurality of true translation pairs to generate a teacher model;
cause the teacher model to generate a plurality of augmented data pairs for each true translation pair of the plurality of true translation pairs, wherein an augmented data pair comprises one sequence from the first domain paired with a model-generated sequence in the second domain and one sequence from the second domain paired with a model-generated sequence in the first domain, wherein the teacher model generates the model-generated sequence in the first domain and the model-generated sequence in the second domain;
create a training dataset comprising the plurality of augmented data pairs and the plurality of true translation pairs;
and
train a bi-encoder with the training dataset and a stochastic gradient descent to distill the teacher model into a trained bi-encoder, wherein the trained bi-encoder generates embeddings for the first domain and embeddings for the second domain, wherein the embeddings for the first domain and the embeddings for the second domain share a joint embedding space.

7 more elements

A computer-implemented method for training a bi-encoder for a neural cross-domain search, comprising:

obtaining a pre-trained sequence-to-sequence deep learning model that produces a conditional-probability distribution to translate a sequence in a first domain into a sequence in a second domain and to translate the sequence in the second domain into the sequence in the first domain, wherein the first domain and the second domain differ, wherein the pre-trained sequence-to-sequence deep learning model comprises at least one encoder and at least one decoder, wherein the sequence in the first domain comprises natural language text and/or source code, wherein the sequence in the second domain comprises natural language text and/or source code;
obtaining a plurality of true translation pairs, wherein a true translation pair of the plurality of true translation pairs comprises a first sequence in the first domain and a second sequence in the second domain;
training the pre-trained sequence-to-sequence deep learning model with the plurality of true translation pairs to generate a teacher model;
causing the teacher model to generate a plurality of augmented data pairs for each true translation pair of the plurality of true translation pairs, wherein an augmented data pair comprises one sequence from the first domain paired with a model-generated sequence in the second domain and one sequence from the second domain paired with a model-generated sequence in the first domain, wherein the teacher model generates the model-generated sequence in the first domain and the model-generated sequence in the second domain;
creating a training dataset comprising the plurality of augmented data pairs and the plurality of true translation pairs;
and
training a bi-encoder with the training dataset and stochastic gradient descent to distill the teacher model into a trained bi-encoder, wherein the trained bi-encoder generates embeddings for the first domain and embeddings for the second domain, wherein the embeddings for the first domain and the embeddings for the second domain share a joint embedding space.

4 more elements

A non-transitory hardware storage device having stored thereon computer executable instructions that are structured to be executable by a processor of a computing device to thereby cause the computing device to train a bi-encoder for a neural cross-domain search by performing actions that:

access a pre-trained sequence-to-sequence deep learning model that produces a conditional-probability distribution to translate a sequence in a first domain into a sequence in a second domain and to translate the sequence in the second domain into the sequence in the first domain, wherein the first domain and the second domain differ, wherein the pre-trained sequence-to-sequence deep learning model comprises at least one encoder and at least one decoder, wherein the sequence in the first domain comprises natural language text and/or source code, wherein the sequence in the second domain comprises natural language text and/or source code;
obtain a plurality of true translation pairs, wherein a true translation pair of the plurality of true translation pairs comprises a first sequence in the first domain and a second sequence in the second domain;
train the pre-trained sequence-to-sequence deep learning model with the plurality of true translation pairs to generate a teacher model;
cause the teacher model to generate a plurality of augmented data pairs for each true translation pair of the plurality of true translation pairs, wherein an augmented data pair comprises one sequence from the first domain paired with a model-generated sequence in the second domain and one sequence from the second domain paired with a model-generated sequence in the first domain, wherein the teacher model generates the model-generated sequence in the first domain and the model-generated sequence in the second domain;
generate a training dataset comprising the plurality of augmented data pairs and the plurality of true translation pairs;
and
train a bi-encoder with the training dataset and stochastic gradient descent to distill the teacher model into a trained bi-encoder, wherein the trained bi-encoder generates embeddings for the first domain and embeddings for the second domain, wherein the embeddings for the first domain and the embeddings for the second domain share a joint embedding space.

4 more elements

Dependencies are inferred from the claim text.

patentgrep

Loading patent…

Abstract

A system for training a bi-encoder for a neural cross-domain search, comprising:

a processor;
and
a memory that stores a program configured to be executed by the processor, wherein the program comprises instructions to perform acts that:
obtain a pre-trained sequence-to-sequence deep learning model that produces a conditional-probability distribution to translate a sequence in a first domain into a sequence in a second domain and to translate the sequence in the second domain into the sequence in the first domain, wherein the first domain and the second domain differ, wherein the pre-trained sequence-to-sequence deep learning model comprises at least one encoder and at least one decoder, wherein the sequence in the first domain comprises natural language text and/or source code, wherein the sequence in the second domain comprises natural language text and/or source code;
obtain a plurality of true translation pairs, wherein a true translation pair of the plurality of true translation pairs comprises a first sequence in the first domain and a second sequence in the second domain;
train the pre-trained sequence-to-sequence deep learning model with the plurality of true translation pairs to generate a teacher model;
cause the teacher model to generate a plurality of augmented data pairs for each true translation pair of the plurality of true translation pairs, wherein an augmented data pair comprises one sequence from the first domain paired with a model-generated sequence in the second domain and one sequence from the second domain paired with a model-generated sequence in the first domain, wherein the teacher model generates the model-generated sequence in the first domain and the model-generated sequence in the second domain;
create a training dataset comprising the plurality of augmented data pairs and the plurality of true translation pairs;
and
train a bi-encoder with the training dataset and a stochastic gradient descent to distill the teacher model into a trained bi-encoder, wherein the trained bi-encoder generates embeddings for the first domain and embeddings for the second domain, wherein the embeddings for the first domain and the embeddings for the second domain share a joint embedding space.

7 more elements