title: >- [Paper Note] Plug-and-Play Attention Linearization for Pretrained Transformers description: >- [ECCV 2026][Object Detection][Efficient Transformer] Training-free linear attention conversion for pretrained ViTs and DETRs via Jacobian-projected normalization and temperature-tuned extrapolation. tags: - ECCV 2026 - Object Detection - Linear Attention - Model Compression - Efficient Transformer date: 2026-09-19 content_hash: f4fa0259ee61187b
Plug-and-Play Attention Linearization for Pretrained Transformers¶
Conference: ECCV 2026
Paper: ECCV Official
Area: Object Detection
Keywords: Linear Attention, Taylor Series Expansion, Training-free Model Compression, Hybrid Transformer Architecture, Plug-and-Play Acceleration
TL;DR¶
This paper introduces LATTE, a training-free attention linearization framework that approximates softmax attention via a second-order Taylor expansion around tunable anchor points paired with Joint Post-Squaring Normalization (JPN), linearizing over half of the attention layers while retaining over 95% of original performance and cutting attention GFLOPs by up to 15×.
Background & Motivation¶
The self-attention mechanism is the foundational pillar behind modern Vision Transformers and multimodal foundation models, delivering exceptional representational capacity by modeling unrestricted pairwise token interactions. However, standard softmax-based attention incurs quadratic computational and memory complexity with respect to sequence length, \(O(N^2d)\). This quadratic scaling creates a prohibitive computational bottleneck when deploying models on high-resolution vision tasks, long sequence understanding, or resource-constrained edge hardware.
Existing acceleration approaches generally focus on kernel-based linear attention (such as Performer, Linear Transformer, and EfficientViT) that decompose the exponential kernel to reduce complexity to \(O(Nd^2)\). Yet these methods suffer from severe representational degradation unless subjected to computationally expensive end-to-end retraining or distillation. While recent second-order Taylor approximations like QT-ViT demonstrated superior accuracy-speed trade-offs when trained from scratch, they rely on a rigid expansion center (typically fixed at zero) that fails to adapt to the diverse dynamic activation statistics across layers and inputs. Furthermore, their decoupled Query and Key post-normalizations induce severe numerical instability when applied post-hoc without extensive fine-tuning.
To bypass the prohibitive cost of fine-tuning massive pretrained models for efficient deployment, this work shifts the paradigm to post-hoc, training-free attention linearization. Core idea: introduce calibration-tuned Taylor expansion points (TTE) coupled with Joint Post-Squaring Normalization (JPN) to locally stabilize quadratic kernel approximations, selecting robust layers via an activation sensitivity score to construct a high-throughput hybrid transformer.
Method¶
Overall Architecture¶
LATTE (Linearized Attention with Tunable Taylor Series Expansion) provides a modular, plug-and-play recipe for converting pretrained vision transformers into linear-softmax hybrid architectures without end-to-end backpropagation. The workflow operates through four synchronized stages: first, target softmax self-attention blocks are converted into a symmetric second-order Taylor feature representation expanded around a tunable anchor \(\alpha\); second, Joint Post-Squaring Normalization (JPN) modulates pairwise similarity scores by their joint energy in squared feature space; third, lightweight scalar parameters (expansion center \(\alpha\), normalization scaling factors \(\delta_q, \delta_k\), and feature augmentation offsets) are calibrated independently per layer on a tiny unlabeled dataset; fourth, a hybrid sensitivity metric combining activation cosine distance and MSE identifies layers most resilient to linearization, reverting sensitive blocks back to standard attention.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Input Tokens X"] --> B["Stage 1: Tunable Taylor Expansion Kernel Mapping<br/>Adaptive anchor alpha and symmetric augmented features"]
B --> C["Stage 2: Joint Post-Squaring Normalization<br/>Pairwise energy inverse modulation and stabilization"]
C --> D["Stage 3: Calibration-Guided Parameter Adaptation<br/>Layer-wise reconstruction on tiny unlabeled data"]
D --> E["Stage 4: Sensitivity-Aware Layer Selection<br/>Cosine distance + MSE ranking for hybrid layout"]
E --> F["Output: High-Throughput Hybrid Transformer"]Key Designs¶
1. Tunable Taylor Expansion Kernel Mapping: Adaptive second-order kernelization Conventional series-based linear attention methods center the exponential Taylor expansion at a predetermined constant, accumulating severe truncation error when input dot-products drift across layers. LATTE expands the softmax kernel \(f(x) = \exp(x)\) (where \(x = \langle \mathbf{q}, \mathbf{k} \rangle / \sqrt{d}\)) around a calibration-tuned, activation-dependent scalar center \(\alpha \in \mathbb{R}\): $\(f(x) \approx e^\alpha \left[ 1 + (x - \alpha) + \frac{1}{2}(x - \alpha)^2 \right] = \frac{e^\alpha}{2} \left[ (x + 1 - \alpha)^2 + 1 \right]\)$ To enable symmetric and efficient linear attention, the authors define \(m = \sqrt{1 - \alpha}\) (with \(1 - \alpha \ge 0\)) and introduce an augmented feature vector \(\mathbf{h}_m(\mathbf{x}) = [\mathbf{x}^\top / d^{1/4}, m]^\top\). Its dot product recovers the inner linear shift: \(\langle \mathbf{h}_m(\mathbf{q}), \mathbf{h}_m(\mathbf{k}) \rangle = \frac{\langle \mathbf{q}, \mathbf{k} \rangle}{\sqrt{d}} + 1 - \alpha\). Expanding the squared inner product via Kronecker products would naively scale the feature space to \(O(d^2)\). To preserve linear attention efficiency, LATTE applies a diagonal Kronecker approximation that retains the self-squared terms and sums identical cross-terms, yielding an effective per-dimension entry \(\frac{x_i^2}{\sqrt{d}} + 2m \frac{x_i}{d^{1/4}} + m^2 = (x_i / d^{1/4} + m)^2\). This collapses the representation into a compact symmetric feature map \(\psi_m(\mathbf{x})\), locking the asymptotic complexity at \(O(Nd^2)\).
2. Joint Post-Squaring Normalization: Damping excessive activations and compressing dynamic range After quadratic feature transformation, extreme token activations can easily destabilize attention distributions in training-free settings. Prior methods normalized queries and keys independently, which discards inter-token geometric interactions. LATTE introduces Joint Post-Squaring Normalization (JPN), scaling squared query and key vectors by their joint \(\ell_2\)-energy: $\(\tilde{\mathbf{q}} = \frac{\delta_q \mathbf{q}^2}{h'(\mathbf{q}, \mathbf{k})}, \quad \tilde{\mathbf{k}} = \frac{\delta_k \mathbf{k}^2}{h'(\mathbf{q}, \mathbf{k})}, \quad \text{where } h'(\mathbf{q}, \mathbf{k}) = \sqrt{(\|\mathbf{q}^2\|_2 + \varepsilon)(\|\mathbf{k}^2\|_2 + \varepsilon)}\)$ The resulting pairwise similarity decomposes into an angular alignment factor and an inverse joint-energy modulator: $\(\text{Sim}(\tilde{\mathbf{q}}, \tilde{\mathbf{k}}) = \frac{\langle \mathbf{q}^2, \mathbf{k}^2 \rangle}{h'(\mathbf{q}, \mathbf{k})} \cdot \frac{\delta_q \delta_k}{h'(\mathbf{q}, \mathbf{k})}\)$ This formulation penalizes token pairs whenever either token possesses abnormally excessive activation magnitude, effectively reproducing the dynamic range compression inherent to softmax and ensuring that dot-products stay within the stable local approximation neighborhood of \(\alpha\).
3. Calibration-Guided Parameter Adaptation: Lightweight local reconstruction and dynamic inference Rather than executing end-to-end training, LATTE optimizes solely a handful of scalar parameters on a small calibration set (e.g., dozens to hundreds of unlabeled images). Each block minimizes a task-agnostic mean-squared activation reconstruction loss, updating only the anchor \(\alpha\), normalization scalars \(\delta_q, \delta_k\), and padding scalars \(\beta_q, \beta_k\) (which absorb the normalizer \(\sum_j \text{Sim}(\mathbf{q}, \mathbf{k}_j)\) into linear matrix multiplication \(\hat{\mathbf{Q}}(\hat{\mathbf{K}}^\top \mathbf{1}_N)\)). For backbones where projection adjustments are needed, re-estimating a lightweight diagonal scaling vector \(\boldsymbol{\sigma} \in \mathbb{R}^d\) on the output projection matrix \(O\) is sufficient to recover performance. This also enables dynamic runtime inference: selecting linear attention for high-resolution, long-sequence inputs (\(N\)) and retaining softmax attention for low-resolution inputs.
4. Sensitivity-Aware Layer Selection: Balancing cosine distance and Frobenius distortion Replacing all attention layers indiscriminately degrades critical high-level representations. LATTE evaluates the empirical sensitivity of each block by computing a hybrid score balancing mean cosine distance and normalized Frobenius MSE between original activations \(\mathbf{A}_\ell\) and linearized activations \(\tilde{\mathbf{A}}_\ell\): $\(\mathcal{I}_\ell = \left( 1 - \frac{1}{BN} \sum_{i=1}^{BN} \frac{\langle \mathbf{a}_{\ell,i}, \tilde{\mathbf{a}}_{\ell,i} \rangle}{\|\mathbf{a}_{\ell,i}\|_2 \|\tilde{\mathbf{a}}_{\ell,i}\|_2 + \varepsilon} \right) + \lambda \cdot \frac{1}{BNd} \|\mathbf{A}_\ell - \tilde{\mathbf{A}}_\ell\|_F^2\)$ Using \(\lambda = 10\), layers are sorted in ascending order of \(\mathcal{I}_\ell\). The top-\(k\) layers with the lowest sensitivity are retained as linearized attention blocks, while the remaining \((L-k)\) layers revert to exact softmax self-attention, maximizing computational speedup while preventing catastrophic representation drift.
Key Experimental Results¶
Main Results¶
LATTE was evaluated across object detection (COCO benchmark on DETR and RT-DETR) and image-text retrieval (MSCOCO Captions on CLIP and SigLip). In all experiments, 67% of the attention layers were linearized under the strict training-free calibration setup.
| Model | Method | Linearized Ratio | Primary Metric 1 ([email protected] / Text Recall) | Primary Metric 2 ([email protected]:0.95 / Image Recall) | Retained Fidelity |
|---|---|---|---|---|---|
| DETR | Exact Self-Attention | 0% | 62.3 ([email protected]) | 42.0 ([email protected]:0.95) | 100% |
| DETR | QT-ViT (baseline) | 67% | 52.5 | 35.9 | 85.5% |
| DETR | LATTE (Ours) | 67% | 59.2 | 39.9 | 95.0% |
| RT-DETR | Exact Self-Attention | 0% | 65.9 ([email protected]) | 49.0 ([email protected]:0.95) | 100% |
| RT-DETR | QT-ViT (baseline) | 67% | 54.5 | 40.6 | 82.9% |
| RT-DETR | LATTE (Ours) | 67% | 62.9 | 46.7 | 95.3% |
| CLIP | Exact Self-Attention | 0% | 79.4 (Text Recall) | 61.1 (Image Recall) | 100% |
| CLIP | QT-ViT (baseline) | 67% | 42.0 | 31.2 | 51.1% |
| CLIP | LATTE (Ours) | 67% | 77.1 | 59.7 | 97.7% |
| SigLip | Exact Self-Attention | 0% | 90.0 (Text Recall) | 76.5 (Image Recall) | 100% |
| SigLip | QT-ViT (baseline) | 67% | 82.3 | 69.0 | 90.2% |
| SigLip | LATTE (Ours) | 67% | 87.9 | 75.8 | 99.1% |
Ablation Study¶
The ablation isolates the individual and joint contributions of Joint Post-Squaring Normalization (JPN) and Tunable Taylor Expansion (TTE) across CLIP and SigLip at a 67% linearization ratio.
| Model | Config | JPN | TTE | Preserved Performance | Text Recall | Image Recall | Note |
|---|---|---|---|---|---|---|---|
| CLIP | Original Pretrained | ✗ | ✗ | 100% | 79.4 | 61.1 | Standard softmax attention |
| CLIP | Base Linearized (QT-ViT) | ✗ | ✗ | 51% | 42.0 | 31.2 | Severe collapse without adaptation |
| CLIP | JPN Only | ✓ | ✗ | 78% | 65.1 | 47.9 | Energy damping restores +27% fidelity |
| CLIP | TTE Only | ✗ | ✓ | 86% | 71.5 | 52.8 | Shifted expansion provides major gains |
| CLIP | Full LATTE (JPN+TTE) | ✓ | ✓ | 98% | 77.1 | 59.7 | Synergistic combination restores 98% |
| SigLip | Original Pretrained | ✗ | ✗ | 100% | 90.0 | 76.5 | Standard baseline |
| SigLip | Base Linearized (QT-ViT) | ✗ | ✗ | 90% | 82.3 | 69.0 | 10% drop in baseline linear mode |
| SigLip | JPN Only | ✓ | ✗ | 91% | 83.2 | 69.6 | Slight improvement |
| SigLip | TTE Only | ✗ | ✓ | 96% | 86.3 | 73.3 | +6% recovery |
| SigLip | Full LATTE (JPN+TTE) | ✓ | ✓ | 99% | 87.9 | 75.8 | Near-lossless recovery |
Key Findings¶
- TTE is the primary driver of representation fidelity: On CLIP, enabling TTE alone increases retained accuracy from 51% to 86%, proving that aligning the expansion anchor with layer activation distributions eliminates severe truncation errors. JPN complements TTE by maintaining stable dynamic range.
- Deep robustness under aggressive layer linearization: In layer scaling sweeps on CLIP (24 layers total), QT-ViT degrades sharply beyond 12 layers (50%), whereas LATTE comfortably linearizes 18 layers (75%) while preserving over 94% of original image and text recall.
- Computational efficiency and latency gains: At sequence length \(T = 1000\), LATTE reduces attention GFLOPs on CLIP by 15× (from 16.384 to 1.082). Under long sequences (\(T = 8192\)), LATTE delivers \(3.2\times \sim 5.8\times\) lower latency compared to optimized FlashAttention kernels.
Highlights & Insights¶
- From static series expansion to distribution-adaptive kernelization: Introducing an input-dependent anchor point \(\alpha\) and formulating symmetric feature maps via diagonal Kronecker aggregation enables accurate second-order approximation in strictly \(O(Nd^2)\) complexity.
- Joint Post-Squaring Normalization mimics Softmax energy damping: Penalizing pairwise attention by their combined \(\ell_2\)-energy prevents outlier tokens from dominating linear attention in post-hoc deployment.
- Extreme sample efficiency and broad cross-task generalization: Calibration on varied subsets of COCO yields near-identical performance, and the scheme transfers smoothly to SugarCrepe, CIFAR-10, EuroSAT, Food101, and language models (RoBERTa).
Limitations & Future Work¶
- Heuristic layer selection limitation: The layer importance metric (Eq. 13) evaluates each block in isolation without modeling inter-layer error propagation; as noted by the authors, this heuristic degrades on self-supervised backbones like DINOv2.
- Short sequence overhead: For very short sequence lengths (\(T \le 128\)), the overhead of explicit feature mapping can overshadow matrix multiplication speedups, making dynamic inference necessary.
- Future directions: Developing architecture-aware global layer search algorithms and exploring piece-wise multi-anchor Taylor expansions for non-convex attention landscapes.
Related Work & Insights¶
- vs QT-ViT: QT-ViT targets training from scratch with fixed zero-centered expansion and decoupled post-normalization; LATTE establishes a training-free paradigm with tunable expansion points and joint normalization, preserving far superior fidelity.
- vs Performer / EfficientViT: Performer suffers from Monte Carlo variance in random feature projections, while EfficientViT requires architectural modifications and retraining; LATTE operates as a drop-in replacement on pretrained weights.
- vs FlashAttention: FlashAttention is bounded by the intrinsic \(O(N^2)\) algorithmic lower bound; LATTE achieves true linear scaling with sequence length, outperforming FlashAttention at long context lengths.
Rating¶
- Novelty: ⭐⭐⭐⭐ [Well-crafted tunable second-order expansion and joint energy normalization for post-hoc linearization]
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Evaluated on detection, multimodal, classification, and language models with comprehensive ablation and latency studies]
- Writing Quality: ⭐⭐⭐⭐⭐ [Clear mathematical derivations and well-structured motivation]
- Value: ⭐⭐⭐⭐⭐ [High practical utility for training-free acceleration on edge and compute-limited hardware]