Back to results
US 2025/0384663 A1Published application
Influential data selection for neural network training
OwnerZhiwei Deng, Tao Li +1inventors
InventorsZhiwei Deng, Tao Li, Yang Li
Life of this patent
Estimating the term…
Abstract
Systems, methods, and apparatus, including computer programs encoded on computer storage media for selecting, from a training data set of training examples, a subset of training examples that will be used for training a neural network by selecting the subset of training examples whose combined gradients over time match a target gradient trajectory.
Claims26
A method performed by one or more computers, the method comprising:
- obtaining a training data set for training a neural network having a plurality of model parameters, the training data set comprising a plurality of training examples;
- obtaining a target data set for evaluating the neural network on a particular set of one or more tasks, the target data set comprising a plurality of target examples;
- obtaining a model parameter trajectory that comprises, for each time point in a sequence of time points during the training of the neural network, respective example values of the model parameters at the time point;
- for each time point in the sequence of time points:
- determining first gradients for the training examples in the training data set with respect to the model parameters at the time point;
- determining a second gradient of the target data set with respect to the model parameters at the time point;
- determining a respective first projection of each of the first gradients onto a respective subspace for the time point;
- determining a second projection of the second gradient for the time point onto the respective subspace for the time point;
- and
- selecting a subset of the training data set as a final training data set for training the neural network by performing an optimization to identify a subset of the training data set that minimizes an error between (i) a target set comprising the second projections for time points and (ii) a training set of weighted first projections for the time points, wherein the weighted first projection for each of the time points is a weighted combination of the respective first projections for the time point in accordance with respective weights for each of the training examples.
7 more elements
A method performed by one or more computers, the method comprising:
- receiving a new network input;
- and
- processing the new network input using a neural network to generate a new network output, wherein the neural network has been trained on a final training data set that has been generated by performing the operations comprising:
- obtaining a training data set for training a neural network having a plurality of model parameters, the training data set comprising a plurality of training examples;
- obtaining a target data set for evaluating the neural network on a particular set of one or more tasks, the target data set comprising a plurality of target examples;
- obtaining a model parameter trajectory that comprises, for each time point in a sequence of time points during the training of the neural network, respective example values of the model parameters at the time point;
- for each time point in the sequence of time points:
- determining first gradients for the training examples in the training data set with respect to the model parameters at the time point;
- determining a second gradient of the target data set with respect to the model parameters at the time point;
- determining a respective first projection of each of the first gradients onto a respective subspace for the time point;
- determining a second projection of the second gradient for the time point onto the respective subspace for the time point;
- and
- selecting a subset of the training data set as a final training data set for training the neural network by performing an optimization to identify a subset of the training data set that minimizes an error between (i) a target set comprising the second projections for time points and (ii) a training set of weighted first projections for the time points, wherein the weighted first projection for each of the time points is a weighted combination of the respective first projections for the time point in accordance with respective weights for each of the training examples.
10 more elements
A system comprising:
- one or more computers;
- and
- one or more storage devices communicatively coupled to the one or more computers, wherein the one or more storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to perform the operations comprising:
- obtaining a training data set for performing instruction tuning of a neural network having a plurality of model parameters, the training data set comprising a plurality of training examples;
- obtaining a target data set for evaluating the neural network on a particular set of one or more instruction following tasks, the target data set comprising a plurality of target examples;
- obtaining a model parameter trajectory that comprises, for each time point in a sequence of time points during the training of the neural network, respective example values of the model parameters at the time point;
- for each time point in the sequence of time points:
- determining first gradients for the training examples in the training data set with respect to the model parameters at the time point;
- determining a second gradient of the target data set with respect to the model parameters at the time point;
- determining a respective first projection of each of the first gradients onto a respective subspace for the time point;
- determining a second projection of the second gradient for the time point onto the respective subspace for the time point;
- and
- selecting a subset of the training data set as a final training data set for training the neural network by performing an optimization to identify a subset of the training data set that minimizes an error between (i) a target set comprising the second projections for time points and (ii) a training set of weighted first projections for the time points, wherein the weighted first projection for each of the time points is a weighted combination of the respective first projections for the time point in accordance with respective weights for each of the training examples.
10 more elements
Dependencies are inferred from the claim text.