ACAM-KD: Making Knowledge Distillation Adaptive | Nebius Science Paper Club

A mask determines which parts of a feature representation receive more weight in the distillation loss. A fixed selection strategy can keep emphasizing regions the student has already learned, while overlooking information it still needs.

Qing Tian illustrates this with an example where a student’s attention initially captures an object, then shifts toward a less useful teacher attention pattern. The example motivates allowing the student’s evolving features to influence knowledge selection instead of relying only on the teacher.

ACAM-KD uses Student-Teacher Cross-Attention Feature Fusion to combine information from both models. The teacher supplies queries, while the student supplies keys and values. The resulting representation reflects the teacher’s features together with the student’s current state.

An Adaptive Spatial-Channel Masking module then generates masks from this fused representation. The masks change throughout training and weight the feature differences used for distillation. The teacher remains frozen; cooperation happens through feature fusion rather than updates to the teacher’s parameters.

Spatial masks weight locations within a feature map. Channel masks weight its feature channels. Combining them lets ACAM-KD select information along both dimensions, and the reported ablation studies show better results than using either type alone.

A diversity loss encourages different selectors to capture complementary information rather than converge on the same pattern. Qing shows masks attending to different parts of a scene, including smaller objects. In the Q&A, he notes that the paper does not include a quantitative ablation isolating the diversity loss.

The paper evaluates object detection on COCO2017 and semantic segmentation on Cityscapes. For detection, it reports improvements of up to 1.4 mAP over the compared state-of-the-art methods when distilling a ResNet-50 student from a ResNet-101 teacher.

For segmentation, it reports a 3.09-point mIoU improvement over the baseline with a DeepLabV3-MobileNetV2 student. These results apply to the evaluated models, datasets and training configurations. They do not establish the same improvement for every teacher-student pairing.

ACAM-KD adds computation during training through cross-attention and mask generation. These additional modules are discarded after distillation, so the deployed student retains its original architecture, parameter count and inference computation.

The benefit is improved accuracy at the existing student’s inference cost. Training overhead still matters: Qing identifies cross-attention as the main additional expense and notes that higher-resolution feature maps can make it more costly. The number of masks is also a manually selected hyperparameter.

Qing proposes extending the same idea to token positions, hidden dimensions and the selection of teacher layers for language models. Multimodal distillation could also adapt how much emphasis different modalities receive.

These are research directions discussed in the session. Qing explicitly confirms that the team had not tested the method in the language domain. The demonstrated ACAM-KD results cover object detection and semantic segmentation, so applications to LLMs, speculative decoding and multimodal models remain to be evaluated.