US 12,602,540 B2Granted

Leveraging a large language model encoder to evaluate predictive models

AssigneeMicrosoft Technology Licensing, LLC

InventorsXiaoqing Wang, Sriram Vasudevan, Shaghayegh Gharghabi, Huichao Xue, Harshul Mulchandani

Life of this patent

Estimating the term…

Loading patent…

Abstract

Aspects of the disclosure include methods for evaluating a predictive model. An exemplary method includes training an evaluation model to output, for an input first entity-second entity pair, a content relevancy prediction. A large language model encoder of the evaluation model generates a first embedding for the first entity and a second embedding for the second entity. The embeddings are fed to an interaction tower to produce a logit and the logit is passed with true labels to a loss function for fine-tuning. The true labels include labeled training data generated by modifying training data having a first proportion of negative labeled data to provide a second proportion of negative labeled data greater than the first proportion. The evaluation model is used to score a performance of a predictive model based at least in part on a comparison of predictions made by the respective models for a same entity pair.

Claims20

Dependent claims

A method comprising:

training an evaluation model to output, for an input entity pair comprising a first entity and a second entity, a prediction comprising a quantitative measurement of a likelihood of a distance between the second entity and the first entity being greater than a predetermined threshold, wherein training the evaluation model comprises:
generating, by a large language model encoder of the evaluation model, a first embedding for the first entity and a second embedding for the second entity;
combining the first embedding with at least one first entity skip connection and combining the second embedding with at least one second entity skip connection, thereby generating a first entity embedding and a second entity embedding, respectively;
feeding the first entity embedding and the second entity embedding to an interaction tower of the evaluation model to produce a logit;
and
passing the logit and one or more true labels to a loss function for fine-tuning, wherein the one or more true labels comprise labeled training data generated by modifying a corpus of training data having a first proportion of negative labeled data such that the training data includes a second proportion of negative labeled data greater than the first proportion;
during an inference phase:
inputting, to the evaluation model, a potential entity pair;
and
outputting, from the evaluation model, a first prediction for the potential entity pair;
during a scoring phase:
receiving, at the evaluation model, a second prediction for the potential entity pair from a predictive model;
and
scoring, by the evaluation model, a performance of the predictive model based at least in part on a comparison of the first prediction to the second prediction;
and
generating, by the evaluation model, evaluation metrics for the predictive model, the evaluation metrics comprising a series of scored predictions for the predictive model over a plurality of predictions of the potential entity pair.

13 more elements

A system having a memory, computer readable instructions, and one or more processors for executing the computer readable instructions, the computer readable instructions controlling the one or more processors to perform operations comprising:

training an evaluation model to output, for an input entity pair comprising a first entity and a second entity, a prediction comprising a quantitative measurement of a likelihood of a distance between the second entity and the first entity being greater than a predetermined threshold, wherein training the evaluation model comprises:
generating, by a large language model encoder of the evaluation model, a first embedding for the first entity and a second embedding for the second entity;
combining the first embedding with at least one first entity skip connection and combining the second embedding with at least one second entity skip connection, thereby generating a first entity embedding and a second entity embedding, respectively;
feeding the first entity embedding and the second entity embedding to an interaction tower of the evaluation model to produce a logit;
and
passing the logit and one or more true labels to a loss function for fine-tuning, wherein the one or more true labels comprise labeled training data generated by modifying a corpus of training data having a first proportion of negative labeled data such that the training data includes a second proportion of negative labeled data greater than the first proportion;
during an inference phase:
inputting, to the evaluation model, a potential entity pair;
and
outputting, from the evaluation model, a first prediction for the potential entity pair;
during a scoring phase:
receiving, at the evaluation model, a second prediction for the potential entity pair from a predictive model;
and
scoring, by the evaluation model, a performance of the predictive model based at least in part on a comparison of the first prediction to the second prediction;
and
generating, by the evaluation model, evaluation metrics for the predictive model, the evaluation metrics comprising a series of scored predictions for the predictive model over a plurality of predictions of the potential entity pair.

13 more elements

A computer program product comprising a non-transitory computer readable storage medium having program instructions embodied therewith, the program instructions executable by one or more processors to cause the one or more processors to perform operations comprising:

training an evaluation model to output, for an input entity pair comprising a first entity and a second entity, a prediction comprising a quantitative measurement of a likelihood of a distance between the second entity and the first entity being greater than a predetermined threshold, wherein training the evaluation model comprises:
generating, by a large language model encoder of the evaluation model, a first embedding for the first entity and a second embedding for the second entity;
combining the first embedding with at least one first entity skip connection and combining the second embedding with at least one second entity skip connection, thereby generating a first entity embedding and a second entity embedding, respectively;
feeding the first entity embedding and the second entity embedding to an interaction tower of the evaluation model to produce a logit;
and
passing the logit and one or more true labels to a loss function for fine-tuning, wherein the one or more true labels comprise labeled training data generated by modifying a corpus of training data having a first proportion of negative labeled data such that the training data includes a second proportion of negative labeled data greater than the first proportion;
during an inference phase:
inputting, to the evaluation model, a potential entity pair;
and
outputting, from the evaluation model, a first prediction for the potential entity pair;
during a scoring phase:
receiving, at the evaluation model, a second prediction for the potential entity pair from a predictive model;
and
scoring, by the evaluation model, a performance of the predictive model based at least in part on a comparison of the first prediction to the second prediction;
and
generating, by the evaluation model, evaluation metrics for the predictive model, the evaluation metrics comprising a series of scored predictions for the predictive model over a plurality of predictions of the potential entity pair.

13 more elements

Dependencies are inferred from the claim text.