System and method for knowledge distillation between neural networks
AssigneeRoyal Bank of Canada
InventorsFrederick Tung, Gregory Mori
Life of this patent
Estimating the term…
Abstract
Systems and methods for knowledge distillation provide supervised training of a student network with a teacher network, including inputting a batch to the teacher network, inputting the batch to the student network, generating a teacher activation map at a layer of the teacher network, generating a student activation map at a layer of the student network corresponding to the layer of the teacher network, generating a pairwise teacher similarity matrix based on the teacher activation map, generating a pairwise student similarity matrix based on the student activation map, and minimizing a knowledge distillation loss defined as a difference between the pairwise teacher similarity matrix and the pairwise student similarity matrix.
Claims20
A knowledge distillation system for supervised training of a computer-implemented student network with a computer-implemented teacher network, the system comprising:
- a processor;
- a memory in communication with the processor, the memory storing instructions that, when executed by the processor cause the processor to:
- input a batch to the teacher network;
- input the batch to the student network;
- generate a teacher activation map at a layer of the teacher network;
- generate a student activation map at a layer of the student network corresponding to the layer of the teacher network;
- generate a pairwise teacher similarity matrix based on the teacher activation map;
- generate a pairwise student similarity matrix based on the student activation map;
- minimize a knowledge distillation loss defined as a difference between the pairwise teacher similarity matrix and the pairwise student similarity matrix, and
- train the student network based on the knowledge distillation loss.
A method for supervised training of a computer-implemented student network with a computer-implemented teacher network, the method comprising:
- inputting a batch to the teacher network;
- inputting the batch to the student network;
- generating a teacher activation map at a layer of the teacher network;
- generating a student activation map at a layer of the student network corresponding to the layer of the teacher network;
- generating a pairwise teacher similarity matrix based on the teacher activation map;
- generating a pairwise student similarity matrix based on the student activation map;
- minimizing a knowledge distillation loss defined as a difference between the pairwise teacher similarity matrix and the pairwise student similarity matrix, and
- training the student network based on the knowledge distillation loss.
Dependencies are inferred from the claim text.