# US 2025/0165801 A1 · Systems and methods for feature dropout knowledge distillation

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2025/0165801 A1
- Document ID: US20250165801A1
- Status: Published application
- Kind: A1
- Application number: 18/954,233
- Filed: 2024-11-20
- Published: 2025-05-22
- Earliest claimed: 2023-11-21
- Assignees: Datum Point Labs Inc.
- Applicants: Datum Point Labs Inc.
- Inventors: Bokyeung Lee; Hanseok Ko; Bonhwa Ku
- Web page: https://patentgrep.com/patent/US20250165801A1

## CPC

- G06N 3/096: Transfer learning
- G06N 3/045: Combinations of networks
- G06N 3/0455: Auto-encoder networks; Encoder-decoder networks
- G06N 3/082: modifying the architecture, e.g. adding, deleting or silencing nodes or connections
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

Embodiments described herein provide systems and methods for knowledge distillation. A system encodes features from a student model and a teacher model to provide principal components. The principal components may be decoded to provide decoded components. Logits may also be output by the student and teacher models. A loss function may be computed based on the principal components, decoded components, and logits. Parameters of the student model may be updated based on the loss function.

## Claims

### Claim 1

A method of knowledge distillation from a teacher model to a student model, comprising:
encoding, via a first encoder, a feature of the student model to provide first principal components;
encoding, via a second encoder, a feature of the teacher model to provide second principal components;
decoding, via a first decoder, the first principal components to provide first decoded components;
decoding, via a second decoder, the second principal components to provide second decoded components;
computing first logits via the student model;
computing second logits from the teacher model;
computing a loss based on at least one of:
a comparison of the first principal components to the second principal components;
a comparison of the first decoded components to the feature of the student model;
a comparison of the second decoded components to the feature of the teacher model; or
a comparison of the first logits to the second logits; and
updating parameters of the student model based on the loss.