Training neural networks on partitioned training data

A system comprising one or more computers and one or more storage devices storing instructions that when executed by one or more computers cause the one or more computers to perform operations comprising:

• obtaining partitioned training data for use in training the neural network, wherein the partitioned training data comprises a plurality of training items each of which is assigned to a respective one of a plurality of partitions;
• obtaining an input specifying a difficulty order for the partitions;
• and
• training the neural network on each of the partitions according to the difficulty order and starting from a first partition in the difficulty order and ending with a last partition in the difficulty order, wherein, for each particular partition of the plurality of partitions, training the neural network comprises:
• generating a sequence of training items for the particular partition, comprising:
• selecting a plurality of training items from among all of the training items in all of the plurality of partitions, and
• generating a sequence of training items that includes the training items selected from among all of the training items in all of the plurality of positions inserted between training items selected only from the plurality of training items in the particular partition;
• and
• training the neural network on the sequence of training items for the partition.

A computer program product encoded on one or more non-transitory computer readable storage media, the computer program product comprising instructions that when executed by one or more computers cause the one or more computers to perform operations comprising:

• obtaining partitioned training data for use in training the neural network, wherein the partitioned training data comprises a plurality of training items each of which is assigned to a respective one of a plurality of partitions;
• obtaining an input specifying a difficulty order for the partitions;
• and
• training the neural network on each of the partitions according to the difficulty order and starting from a first partition in the difficulty order and ending with a last partition in the difficulty order, wherein, for each particular partition of the plurality of partitions, training the neural network comprises:
• generating a sequence of training items for the particular partition, comprising:
• selecting a plurality of training items from among all of the training items in all of the plurality of partitions, and
• generating a sequence of training items that includes the training items selected from among all of the training items in all of the plurality of positions inserted between training items selected only from the plurality of training items in the particular partition;
• and
• training the neural network on the sequence of training items for the partition.