Authority-based training process for a large language model

A computer-implemented method comprising:

• for each sample in a training dataset for training a large language model, generating corresponding sets of data quality metrics, wherein the training dataset encompasses a plurality of topics;
• for each sample in the training dataset, generating a corresponding set of authority scores based on the corresponding sets of data quality metrics for the sample, each authority score in the corresponding set of authority scores indicating a respective authority level of the sample in relation to a particular topic of the plurality of topics;
• and
• executing a training process in which the large language model is trained using a loss function that includes a set of weights, wherein the training process involves dynamically adjusting the set of weights based on the corresponding set of authority scores for each sample in the training dataset.

A non-transitory computer-readable medium comprising program code that is executable by one or more processors for causing the one or more processors to perform operations including:

• for each sample in a training dataset for training a large language model, generating corresponding sets of data quality metrics, wherein the training dataset encompasses a plurality of topics;
• for each sample in the training dataset, generating a corresponding set of authority scores based on the corresponding sets of data quality metrics for the sample, each authority score in the corresponding set of authority scores indicating a respective authority level of the sample in relation to a particular topic of the plurality of topics;
• and
• executing a training process in which the large language model is trained using a loss function that includes a set of weights, wherein the training process involves dynamically adjusting the set of weights based on the corresponding set of authority scores for each sample in the training dataset.