Regularizing machine learning models

A system for training a neural network, wherein the neural network is configured to receive an input data item and to process the input data item to generate a respective score for each label in a predetermined set of multiple labels, the system comprising:

• one or more data processing apparatus;
• and
• one or more memory devices storing instructions that when executed by the one or more data processing apparatus cause the one or more data processing apparatus to perform operations for training a neural network, wherein the neural network is configured to receive an input data item and to process the input data item to generate a respective score for each label in a predetermined set of multiple labels, the operations including:
• obtaining a plurality of training items, wherein each training item is associated with an initial target label distribution that assigns a respective target score to each label in the predetermined set of labels;
• for each training item, modifying the initial target label distribution by combining the initial target label distribution with a smoothing label distribution, wherein the smoothing label distribution includes a respective smoothing score for each label in the predetermined set of multiple labels and is independent of the training item;
• and
• training the neural network using the plurality of training items and the respective modified initial target distributions for the plurality of training items.

A non-transitory computer-readable medium storing software comprising instructions executable by one or more computers which, upon such execution, cause the one or more computers to perform operations for training a neural network, wherein the neural network is configured to receive an input data item and to process the input data item to generate a respective score for each label in a predetermined set of multiple labels, the operations comprising:

• obtaining a plurality of training items, wherein each training item is associated with an initial target label distribution that assigns a respective target score to each label in the predetermined set of labels;
• for each training item, modifying the initial target label distribution by combining the initial target label distribution with a smoothing label distribution, wherein the smoothing label distribution includes a respective smoothing score for each label in the predetermined set of multiple labels and is independent of the training item;
• and
• training the neural network using the plurality of training items and the respective modified initial target distributions for the plurality of training items.

A method of training a neural network, wherein the neural network is configured to receive an input data item and to process the input data item to generate a respective score for each label in a predetermined set of multiple labels, the method comprising:

• obtaining a set of training data that includes a plurality of training items, wherein each training item is associated with a respective label from the predetermined set of multiple labels;
• modifying the set of training data, comprising, for each training item:
• determining whether or not to modify the label associated with the training item;
• and
• in response to determining to modify the label associated with the training item, changing the label associated with the training item that correctly describes the training item to a different label from the predetermined set of labels that incorrectly describes the training item;
• and
• training the neural network on the modified training data.

A method of training a neural network, wherein the neural network is configured to receive an input data item and to process the input data item to generate a respective score for each label in a predetermined set of multiple labels, the method comprising:

• receiving a request to train the neural network to optimize a loss function comprising a first error term;
• and
• training the neural network to optimize a regularized loss function, the regularized loss function comprising the first error term and a regularizing error term that penalizes the neural network based on the error between a predicted distribution and a smoothing distribution, wherein:
• the predicted distribution is generated by the neural network upon processing a particular data item and includes a respective predicted score for each label in the predetermined set of labels;
• and
• the smoothing distribution includes a respective smoothing score for each label in the predetermined set of multiple labels.