Towards Long-window Anchoring in Vision-Language Model Distillation

Conference: AAAI 2026 arXiv: 2512.21576 Code: None Area: Multimodal VLM Keywords: Knowledge Distillation, Long-context VLM, RoPE Positional Encoding, Fourier Analysis, Context Window Extension

TL;DR

LAid (Long-window Anchoring distillation) proposes a position-aware knowledge distillation framework that extends the effective context window of small VLMs (3B/7B) to 3.2× their original size—approaching the level of a large teacher model (32B)—through head-level Fourier-enhanced positional knowledge transfer, while preserving performance on standard VL benchmarks.

Background & Motivation

State of the Field

Large VLMs (≥72B parameters) have demonstrated context window capabilities of up to 128K tokens (e.g., Gemma 3, Qwen 2.5-VL, InternVL 3). However, their distilled/smaller counterparts (≤7B parameters), despite sharing identical positional encodings, architectures, and training recipes, suffer from a significant context window shrinkage. This issue is negligible in short-context evaluations but becomes a primary bottleneck in full-length inference scenarios.

Limitations of Prior Work

Window shrinkage in small models: This paper is the first to identify the phenomenon of "window shrinkage" in distilled VLM branches. Experiments show that Qwen2.5-VL-3B degrades in performance on multi-image tasks at 5.2× the rate of the 32B model, even when both perform comparably on short contexts.

Failure of conventional context extension methods: Positional encoding extrapolation methods (YaRN, SelfExtend) are effective for text-only LLMs but lead to performance degradation when directly applied to VLMs. This is because VLMs' multimodal nature—visual tokens introducing densely structured spatial information—violates the assumptions underlying text-based extension methods.

Frequency leakage: From a signal-processing perspective, RoPE encodes positional information using a set of predefined frequencies forming a truncated Fourier series. Small models, with limited capacity, cannot represent the full frequency spectrum, leading to frequency leakage and distortion that causes attention to decay rapidly over long distances.

Core Insight

Knowledge distillation can inadvertently improve a student model's responsiveness to RoPE. Standard distillation, while not explicitly optimizing for long-context capability, incidentally enhances the student's positional encoding awareness. The core idea of LAid is to transform this incidental effect into a systematic optimization objective.

Starting Point

The paper introduces the concept of Long-window Anchoring: rather than extending context from the model's own perspective (as in traditional approaches), a large model serves as an "anchor," and post-training aligns the small model's long-window capability to the large model's level. Among various post-training strategies, knowledge distillation is adopted as the foundation and is specifically optimized for positional knowledge transfer.

Method

Overall Architecture

The LAid framework comprises two complementary components: (1) progressive distance-weighted attention matching, which dynamically emphasizes long-range positional differences during training; and (2) learnable RoPE response gain modulation, which selectively amplifies the required positional sensitivity. The core mechanism is head-level Fourier-enhanced positional distillation for transferring positional knowledge from teacher to student.

Key Designs

1. Head-level Position Alignment

• Function: Trains each attention head of the student model to learn a weighted combination of multiple attention heads from the teacher model.
• Mechanism: For student layer \(l\) head \(i\) and teacher layer \(L\) head set \(\{j\}\), weights \(\{w_{i,j}\}\) are learned such that: $\(Q_{l,i}^s \approx \sum_{j=1}^{h_t} w_{i,j} \cdot Q_{L,j}^t, \quad K_{l,i}^s \approx \sum_{j=1}^{h_t} w_{i,j} \cdot K_{L,j}^t\)$ Each student head learns not from a single teacher head but from a weighted combination of multiple teacher heads, where \(w_{i,j}\) determines each teacher head's contribution.
• Design Motivation: Research indicates that certain "positional heads" in large models specialize in modeling long-range dependencies while "local heads" focus on short-range interactions. Through weighted combination, a student head can simultaneously inherit both local and global positional awareness.

2. Fourier-Enhanced Position Distillation

• Function: Understands and optimizes positional knowledge transfer from a signal-processing perspective.
• Mechanism: When RoPE is applied to the teacher's Q and K representations, the distillation process learns a linear combination of frequency-encoded representations: $\(Q_{l,i,rot}^s \approx \sum_{j=1}^{h_t} w_{i,j} \cdot (Q_{L,j}^t \odot R_\theta(m))\)$ This is equivalent to learning an enhanced rotary encoding: $\(R'_\theta(m) = \sum_{j=1}^{h_t} w_{i,j} \cdot (W_{t,j}^Q \cdot R_\theta(m) \cdot (W_{t,j}^Q)^{-1})\)$ The key implication is that the student no longer learns the limited frequency set of standard RoPE but instead acquires a richer Fourier series representation, breaking the frequency constraints of standard RoPE and mitigating leakage and distortion.
• Design Motivation: Small models suffer from RoPE frequency leakage—the inability to represent all necessary frequencies causes long-range attention decay. By combining teacher heads via weighted aggregation, the representable frequency range is indirectly expanded.

3. Complete Distillation Objective

• Function: Combines the positional distillation loss with conventional distillation losses.
• Core Formula: $\(\mathcal{L}_{total} = \lambda_{LAid} \cdot \mathcal{L}_{LAid} + \lambda_{KL} \cdot \mathcal{L}_{KL} + \lambda_{SFT} \cdot \mathcal{L}_{SFT}\)$
• \(\mathcal{L}_{LAid}\): Frobenius norm alignment loss on head-level Q/K matrices
• \(\mathcal{L}_{KL}\): KL divergence between teacher and student output distributions (with temperature \(\tau\))
• \(\mathcal{L}_{SFT}\): Standard supervised fine-tuning loss
• Note: \(\mathcal{L}_{KL}\) is excluded when vocabulary sizes of the student (3B, vocab=151936) and teacher (32B, vocab=152064) are mismatched.

Loss & Training

• Teacher: Qwen2.5-VL-32B-Instruct
• Student: Qwen2.5-VL-7B-Instruct / 3B-Instruct
• Optimizer: AdamW (model parameters lr=1e-5, weight coefficients lr=1e-4)
• Training: 10 epochs, 4× NVIDIA A800
• Batch: per-device=1, gradient accumulation=8, effective batch size=8
• Alignment applied only to the last layer (7B→Layer 27, 3B→Layer 35)
• Training data: 5,000 QA pairs from the Visual HayStacks training set, haystack size 2–20 images
• Training time: ~74 hours for 7B, ~43 hours for 3B

Key Experimental Results

Main Results (Visual HayStack Benchmark)

Params	Method	1img	5img	10img	20img	50img	100img	150img	Short-ctx Gain	Long-ctx Gain
32B	Baseline	83.79	79.11	74.71	73.34	68.17	62.56	60.65	-	-
7B	Baseline	80.22	68.45	62.19	57.21	54.73	51.08	47.43	-	-
7B	YaRN	80.03	63.78	62.09	55.96	56.26	47.96	42.36	-2.5%	-4.7%
7B	SelfExtend	78.53	62.58	58.69	53.01	50.35	45.12	40.12	-5.9%	-11.7%
7B	SFT(LoRA)	97.78	92.92	85.80	84.73	63.10	52.28	43.08	+35.9%	+3.6%
7B	LAid	92.83	83.26	80.46	74.09	67.04	63.37	60.17	+24.1%	+24.5%
3B	Baseline	85.91	65.70	62.09	52.16	50.22	47.80	41.67	-	-
3B	LAid	96.83	83.34	74.29	63.27	58.20	53.91	50.23	+20.1%	+16.4%

Ablation Study (7B Model)

Configuration	1img	10img	50img	100img	Short-ctx Gain	Long-ctx Gain
Baseline	80.22	62.19	54.73	47.43	-	-
w/o \(\mathcal{L}_{KL}\)	91.26	75.09	66.11	62.29	+18.2%	+20.2%
w/o \(\mathcal{L}_{LAid}\)	87.68	72.57	64.61	61.50	+15.5%	+18.1%
Full LAid	92.83	80.46	67.04	63.37	+24.1%	+24.5%

Key Findings

1. Conventional context extension methods fail on VLMs: YaRN degrades long-context performance by 4.7% and SelfExtend by 11.7%. The multimodal specificity of VLMs—spatial organization of visual tokens and cross-modal alignment—renders text-only methods inapplicable.
2. SFT exhibits a short-context bias: SFT yields a remarkable short-context gain (+35.9%) but only +3.6% on long contexts, indicating overfitting to short-context patterns with no generalization to long sequences.
3. LAid achieves balanced long- and short-context improvement: +24.1% on short contexts and +24.5% on long contexts, making it the only method that substantially improves both dimensions.
4. 7B LAid achieves 63.37% at 100 images, surpassing the 32B baseline (62.56%): A small model surpasses its large counterpart on long-context tasks through distillation, demonstrating that positional knowledge can be effectively transferred.
5. \(\mathcal{L}_{LAid}\) is the core contribution: Its removal causes a short-context drop of 8.6% and a long-context drop of 6.4%, confirming that positional alignment is critical.
6. Spectral analysis validates the approach: LAid successfully preserves critical low-frequency attention components (global positional heads), which conventional methods fail to transfer.

Highlights & Insights

1. Problem identification as a contribution: This paper is the first to systematically identify "window shrinkage" in small VLMs and to quantify the gap via Visual HayStack experiments (3B degrades 5.2× faster than 32B).
2. Deep Fourier-perspective analysis: Interpreting RoPE as a truncated Fourier series and linking frequency leakage to long-context performance degradation provides a principled theoretical basis for the distillation objective design.
3. Clear definition of Long-window Anchoring: The paradigm is clearly distinguished from conventional context extension, establishing a post-training alignment framework that uses large models as anchors.
4. Intuitive head-level knowledge flow analysis: Visualizations illustrate the division of labor between "local heads" and "global heads" in the teacher, and how LAid transfers this specialization to the student.
5. Key insight in experimental design: Training only on short-context data (2–20 images) enables the model to extrapolate to 150-image long contexts, demonstrating that positional awareness is a generalizable capability.

Limitations & Future Work

1. Focus limited to attention layers: The role of feed-forward networks (FFN) in long-context modeling is not considered, potentially missing part of the positional information.
2. Distillation overhead: Although inference is unaffected, training requires 74 hours (7B) and necessitates loading both teacher and student models simultaneously.
3. Validation on a single benchmark: Visual HayStack is the only long-context VL evaluation benchmark used; validation on other tasks (e.g., long document understanding, multi-turn dialogue) is absent.
4. Limited to the Qwen2.5-VL family: Applicability to other VLM architectures such as InternVL and LLaVA remains unknown.
5. Fixed LoRA rank of 8: The SFT baseline may achieve better results with a larger rank, potentially making the comparison not entirely fair.
6. Only the last layer is aligned: Whether aligning intermediate layers would yield additional gains is unexplored.

Related Work & Insights

• YaRN / SelfExtend: Conventional RoPE extension methods, which this paper demonstrates fail on VLMs, underscoring the need for multimodal-specific solutions.
• LongReD: Explores distillation for long-context RoPE in LLMs but does not address VLMs.
• M-RoPE (Multimodal RoPE): The multimodal positional encoding used in Qwen2.5-VL, integrating positional information across text, images, and video.
• Insight: Knowledge transfer between models encompasses not only semantic-level information (logit distillation) but also structural knowledge (positional encoding response), the latter of which may require dedicated distillation objectives.

Rating

• Novelty: ⭐⭐⭐⭐⭐ (Novel problem formulation, unique Fourier-analysis perspective, pioneering Long-window Anchoring concept)
• Experimental Thoroughness: ⭐⭐⭐⭐ (Complete main experiments, ablations, comparisons, and visualizations, though limited to one benchmark and one model family)
• Writing Quality: ⭐⭐⭐⭐⭐ (Theoretical derivations are clear; the logical chain from problem identification to method design is coherent)
• Value: ⭐⭐⭐⭐⭐ (Addresses a critical practical deployment issue; enabling 3B/7B models to achieve long-context capability approaching 32B has high application value)

Related Papers

• [AAAI 2026] EM-KD: Distilling Efficient Multimodal Large Language Model with Unbalanced Vision Tokens
• [AAAI 2026] Few-Shot Precise Event Spotting via Unified Multi-Entity Graph and Distillation
• [CVPR 2026] ReMoRa: Multimodal Large Language Model based on Refined Motion Representation for Long-Video Understanding
• [AAAI 2026] LLMC+: Benchmarking Vision-Language Model Compression with a Plug-and-play Toolkit
• [CVPR 2026] Uncertainty-Aware Knowledge Distillation for Multimodal Large Language Models