Methods, devices and media providing an integrated teacher-student system
AssigneeHuawei Technologies Co., Ltd.
InventorsDeepak Sridhar, Juwei Lu
Life of this patent
Estimating the term…
Abstract
Methods, devices and processor-readable media for an integrated teacher-student machine learning system. One or more teacher-student modules are trained as part of the teacher neural network training. Each student sub-network uses a portion of the teacher neural network to generate an intermediate feature map, then provides the intermediate feature map to a student sub-network to generate inferences. The student sub-network may use a feature enhancement block to map the intermediate feature map to a subsequent feature map. A compression block may be used to compress intermediate feature map data for transmission in some embodiments.
Claims20
A method, comprising:
- operating a first teacher sub-network of a teacher neural network of an integrated system in an inference mode to generate a first intermediate feature map based on an input data received by the integrated system;
- and
- operating a first student sub-network of the integrated system in the inference mode to generate student inference data based on the first intermediate feature map,
- wherein:
- the first student sub-network has been trained when operating in a training mode, using subsequent inference data provided as a supervision signal, the subsequent inference data being generated by a second student sub-network of the integrated system that is trained using teacher inference data generated by the teacher neural network.
A method, comprising:
- receiving a feature map;
- generating positional normalization data based on the feature map;
- down-sampling the feature map to generate an embedding vector;
- generating compressed data comprising the positional normalization data and the embedding vector;
- and
- transmitting the compressed data over a communication link.
A method, comprising:
- providing an integrated system comprising:
- a teacher neural network comprising, in series:
- a first teacher sub-network adapted to generate a first intermediate feature map based on a training-mode data input to the teacher neural network;
- and
- a final teacher sub-network adapted to generate teacher inference data based on the first intermediate feature map;
- and
- a first student sub-network adapted to generate first student inference data based on the first intermediate feature map;
- and
- training the integrated system by:
- propagating the training-mode data input forward through the teacher neural network to generate the first intermediate feature map and teacher inference data;
- propagating the first intermediate feature map forward through the first student sub-network to generate first student inference data;
- calculating a first knowledge distillation loss based on a knowledge distillation loss function applied to the first student inference data and the teacher inference data;
- and
- propagating the first knowledge distillation loss backward through the first student sub-network and the first teacher sub-network to train the first student sub-network and the first teacher sub-network.
Dependencies are inferred from the claim text.