Knowledge distillation for neural networks using multiple augmentation strategies

A non-transitory computer readable medium comprising instructions that, when executed by at least one processor, cause a computing device to:

• sample a first augmentation range to select augmentation measures to apply to a set of digital images, wherein the first augmentation range comprises a first upper limit;
• generate a first set of augmented digital images by augmenting the set of digital images using a first augmentation strategy to apply the augmentation measures selected from the first augmentation range;
• learn parameters of a source neural network based on comparing classifications of the first set of augmented digital images with ground truth labels;
• generate a second set of augmented digital images by augmenting the set of digital images using a second augmentation strategy to apply augmentation measures selected from a second augmentation range with a second upper limit greater than the first upper limit;
• and
• learn parameters of a distilled neural network from the parameters learned for the source neural network utilizing the second set of augmented digital images based on comparing a first set of classifications of the second set of augmented digital images generated by the source neural network with a second set of classifications of the second set of augmented digital images generated by the distilled neural network.

A system comprising:

• one or more memory devices comprising a set of digital images, a source neural network, and a distilled neural network;
• and
• one or more computing devices that are configured to cause the system to:
• sample a light augmentation range to select augmentation measures to apply to the set of digital images, wherein the light augmentation range comprises a first upper limit
• generate lightly augmented digital images by augmenting the set of digital images using a light augmentation strategy to apply the augmentation measures selected from the light augmentation range;
• learn parameters for the source neural network utilizing the lightly augmented digital images;
• and
• transfer learned parameters from the source neural network to the distilled neural network by:
• generating heavily augmented digital images by augmenting the set of digital images using a heavy augmentation strategy to apply augmentation measures selected from a heavy augmentation range with a second upper limit greater than the first upper limit;
• determining a first set of classifications utilizing the source neural network and a second set of classifications utilizing the distilled neural network from the heavily augmented digital images;
• and
• learning parameters of the distilled neural network based on comparing the first set of classifications and the second set of classifications.