Back to results
US 2025/0165801 A1Published application
Systems and methods for feature dropout knowledge distillation
AssigneeDatum Point Labs Inc.
InventorsBokyeung Lee, Hanseok Ko, Bonhwa Ku
Life of this patent
Estimating the term…
Abstract
Embodiments described herein provide systems and methods for knowledge distillation. A system encodes features from a student model and a teacher model to provide principal components. The principal components may be decoded to provide decoded components. Logits may also be output by the student and teacher models. A loss function may be computed based on the principal components, decoded components, and logits. Parameters of the student model may be updated based on the loss function.
Claims1
A method of knowledge distillation from a teacher model to a student model, comprising:
- encoding, via a first encoder, a feature of the student model to provide first principal components;
- encoding, via a second encoder, a feature of the teacher model to provide second principal components;
- decoding, via a first decoder, the first principal components to provide first decoded components;
- decoding, via a second decoder, the second principal components to provide second decoded components;
- computing first logits via the student model;
- computing second logits from the teacher model;
- computing a loss based on at least one of:
- a comparison of the first principal components to the second principal components;
- a comparison of the first decoded components to the feature of the student model;
- a comparison of the second decoded components to the feature of the teacher model;
- or
- a comparison of the first logits to the second logits;
- and
- updating parameters of the student model based on the loss.
11 more elements
Dependencies are inferred from the claim text.