Knowledge distillation with adaptive frequency prompting

This paper proposes an enhanced frequency-domain knowledge distillation framework to address limitations in spatial-domain approaches, where multiple downsampling operations compromise detail preservation and conventional attention-based mechanisms fail to fully capture the global contextual information. (1) An adaptive frequency prompt module where the frequency prompt interacts with teacher frequency bands during fine-tuning to capture contextual semantic frequency. During the distillation process, the frequency prompt is used to generate a pixel-by-pixel mask to locate the pixels of interest in different frequency bands. The channel-level position-sensitive weight is designed to provide high-order spatial enhancement. (2) A feature fusion module that hierarchically fuses multilevel features to reinforce the local structure. (3) Extensive experiments demonstrate state-of-the-art performance, when the teacher-student architecture is the same, achieving 1.83% and 1.03% Top-1 accuracy improvements over ReviewKD and CAT-KD on the CIFAR-100 dataset, and it also performs competitively on the Tiny-ImageNet dataset, along with a 4.5% average precision improvement for the anchor-free detector FCOS-R50 on the MS COCO dataset. The framework's effectiveness is further validated through cross-architecture evaluations, showing consistent superiority in balancing model efficiency and accuracy. This work provides new insights into frequency-aware knowledge distillation for lightweight model optimization.