Dissecting Transformers: A CLEAR Perspective Towards Green AI¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/divyanash911/CLEAR
Area: Interpretability
Keywords: Green AI, Mechanistic Interpretability, Energy Profiling, Transformer Components, Amplification Strategy
TL;DR¶
Addressing the severe temporal mismatch between microsecond-scale component execution and millisecond-scale hardware energy sensors, this work introduces CLEAR—a three-stage profiling framework using activation caching and repetition amplification—to conduct the first comprehensive component-level inference energy dissection across 15 models and 4 Transformer architectures.
Background & Motivation¶
The ubiquitous deployment of large language models across consumer and enterprise services has escalated serious sustainability and carbon footprint concerns. For instance, serving ChatGPT-4o was projected to emit roughly 150,000 tons of CO2e in 2025 alone, comparable to the annual emissions of tens of thousands of gasoline-powered vehicles. Unlike model pre-training, which constitutes an infrequent fixed expenditure, online inference operates continuously at massive concurrency, serving hundreds of millions of user queries daily and dominating the cumulative lifetime energy footprint of AI systems. Nevertheless, existing sustainability literature treats models as black boxes, reporting only coarse-grained, aggregate energy metrics over massive evaluation workloads (often at the megajoule scale), which systematically obscures the true energetic contribution of internal architectural primitives.
The fundamental technical obstacle to component-level energy attribution lies in the temporal granularity mismatch between execution timescales and sensor monitoring intervals. Individual Transformer primitives—such as Attention sub-blocks or normalization layers—typically complete execution on modern GPUs within 10 to 100 microseconds. In contrast, onboard hardware power interfaces (such as NVIDIA NVML) update readings only once every 20 to 50 milliseconds. Consequently, naive sampling before and after a single operator call either returns an erroneous zero reading because execution concluded entirely between consecutive sensor polling intervals, or produces measurements dominated by baseline CUDA idle energy noise. Furthermore, hardware-intrusive execution tracing methods depend on specialized physical instrumentation that cannot scale across diverse compute environments, leaving the community without an accessible, verified software-level profiling methodology.
To overcome this impasse and provide verifiable ground truth for sustainable model design, this paper proposes CLEAR (Component-Level Energy Assessment via Repetitions). The framework intercepts input activations via forward hooks and replays candidate components in rapid, gapless succession, expanding execution duration to hundreds of milliseconds so that computational energy reliably swamps sensor quantization noise, backed by rigorous consistency and completeness validation. Core idea: by combining forward activation caching with gapless repetition amplification, microsecond-scale component workloads are stretched into the valid sampling window of standard hardware power sensors, establishing a purely software-based, dual-validated fine-grained energy profiling framework.
Method¶
Overall Architecture¶
CLEAR provides a lightweight, hardware-agnostic three-stage pipeline comprising an Activation Store, an Amplification Strategy, and a dual Validation procedure. The methodology systematically decomposes the Transformer architecture into its constituent computational building blocks: Attention modules, feed-forward multilayer perceptrons (MLP), normalization layers (Norm), embedding layers, and the output language modeling head (LM Head).
The end-to-end execution flow proceeds through three sequential phases, as visualized below. First, forward hooks intercept and cache true operational activation tensors. Next, an amplification loop executes consecutive forward invocations of the isolated primitive without intervening stalls, ensuring the aggregated workload spans hundreds of milliseconds to exceed the 20–50 ms NVML polling period. Finally, energy readings are verified for statistical consistency across trials and energy conservation across blocks and the entire model.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Inference Input & Computation Graph"] --> B["Stage 1: Forward Activation Store<br/>Forward hooks intercept and cache tensor ac"]
B --> C["Stage 2: Gapless Repetition Amplification<br/>N back-to-back executions suppress sensor noise"]
C --> D["Stage 3: Dual Consistency & Completeness Validation<br/>Multi-trial variance check and full-model energy balance"]
D --> E["Component-Level Energy & FLOP Profiling"]
Key Designs¶
1. Forward Activation Store: Preserving Input Distribution Under Isolated Replay To profile internal components in strict isolation without altering input data statistics or inducing synthetic distribution drift, forward hooks are registered across critical boundaries of the PyTorch computation graph. During a standard forward pass, the exact input activation tensor \(a_c \in \mathbb{R}^d\) entering component \(c\) is captured and recorded into cache store \(\mathcal{A} = \{a_c \mid c \in \mathcal{A}\}\). Subsequent benchmarking feeds this cached representation directly into the isolated component, ensuring zero runtime gradient overhead, no spurious memory reallocation artifacts, and complete behavioral fidelity relative to an end-to-end forward pass.
2. Gapless Repetition Amplification: Bridging the Temporal Sensor Gap Because individual primitive latency (10–100 \(\mu\text{s}\)) is orders of magnitude shorter than the NVML sensor polling interval (20–50 \(\text{ms}\)), single-call energy readouts \(\Delta E\) are either falsely reported as zero or corrupted by baseline CUDA idle energy \(\varepsilon\). CLEAR overcomes this by repeatedly invoking component \(c\) on cached input \(a_c\) for \(N\) consecutive executions without idle gaps. The per-invocation energy is derived as: $$ \hat{E}c = \frac{E $$ Setting }} - E_{\text{start}}}{N} = \frac{E_{\text{tot}}^c}{N} \pm \frac{\varepsilon}{N\(N = 10,000\) for microsecond-scale sub-blocks and \(N = 1,000\) for full models scales total execution duration to hundreds of milliseconds, ensuring the workload dominates background consumption while suppressing the noise term \(\varepsilon / N \to 0\). Measurements are repeated across \(T=20\) independent trials with sensor cool-down intervals to yield stable mean and standard deviation metrics.
3. Dual Consistency and Completeness Validation: Enforcing Software Energy Conservation To verify that component-wise attribution reflects physical reality rather than profiling artifacts, CLEAR enforces two rigorous validation criteria. Consistency requires that the relative standard deviation across \(T=20\) trials remains below \(9.5\%\) for components consuming \(>5\,\text{mJ}\). Completeness establishes an energy conservation audit by summing individual component measurements into Captured Energy and computing the ratio against end-to-end measured energy: $$ \text{Capture} = \sum_{i=1}^M \bar{E}i, \quad \%\text{Capture} = \left(\frac{\sum}^M \bar{Ei}{\bar{E}\right) \times 100\% $$ Empirical results confirm that block-level capture rates achieve }}\(\% \text{Capture}(\text{Block}) \approx 100\%\) and model-level capture rates satisfy \(\% \text{Capture}(\text{Model}) \ge 90\%\), confirming that virtually all active energy—excluding negligible residual vector additions—is accounted for.
Loss & Training¶
This study focuses strictly on inference-time energy profiling of pre-trained and instruction-tuned checkpoints, involving no parameter updates or training loss optimization. Experiments benchmark FP32, FP16, and BF16 numerical precisions across NVIDIA RTX 6000 Ada, RTX 5000 Ada, and Hopper H100 platforms, systematically varying input sequence length (1–512 tokens), batch size (1–64), attention head counts, and hidden dimensionality.
Key Experimental Results¶
Main Results¶
CLEAR was validated across 15 models spanning Encoder-only (BERT, RoBERTa, ALBERT, DistilBERT), Decoder-only (Qwen2.5-3B, Llama-3.2-3B, Gemma-3-4B, Phi-4-mini), and MoE (GPT-OSS-20B) families. The table below presents component-wise energy breakdowns and validation completeness for GPT-OSS-20B on the NVIDIA RTX 6000 Ada GPU across varying input sequence lengths.
| Component / Metric | 8 Tokens (mJ) | 32 Tokens (mJ) | 64 Tokens (mJ) | 96 Tokens (mJ) | 128 Tokens (mJ) |
|---|---|---|---|---|---|
| Attention Block (Avg ± Std) | 53.26 ± 1.68 | 64.15 ± 0.69 | 75.16 ± 0.76 | 93.91 ± 0.78 | 100.70 ± 1.05 |
| MLP (Avg ± Std) | 685.41 ± 12.61 | 776.91 ± 3.17 | 867.69 ± 0.87 | 958.13 ± 1.41 | 1046.20 ± 1.19 |
| Normalization (All) | 9.32 ± 0.73 | 10.79 ± 0.83 | 12.70 ± 1.06 | 13.44 ± 1.42 | 14.64 ± 1.11 |
| Measured Block (Avg ± Std) | 731.91 ± 12.46 | 856.31 ± 1.43 | 951.87 ± 0.81 | 1057.01 ± 0.88 | 1157.20 ± 1.18 |
| Block Capture (%Capture) | 102.20% | 99.48% | 100.39% | 100.80% | 100.38% |
| LM Head (Avg ± Std) | 443.39 ± 1.11 | 452.14 ± 0.99 | 460.38 ± 0.99 | 475.22 ± 1.27 | 483.52 ± 1.24 |
| Measured Model (Avg ± Std) | 18447.50 ± 63.78 | 21366.69 ± 103.48 | 24126.47 ± 12.67 | 26634.05 ± 15.33 | 28801.98 ± 2.87 |
| Model Capture (%Capture) | 97.65% | 98.33% | 96.63% | 97.06% | 98.13% |
Ablation Study¶
To evaluate the impact of algorithmic attention variants and graph compilation strategies, the table below compares energy consumption, computational operations, and energy efficiency (\(E/\text{FLOP}\)) on Qwen2.5-3B and Gemma-3-4B at 256 input tokens.
| Model Config (256 Tokens) | Implementation / Compilation Mode | Energy (mJ) | Std Dev (mJ) | FLOPs (GFLOPs) | Energy / GFLOP (mJ) |
|---|---|---|---|---|---|
| Qwen2.5-3B | FP16 Eager Attention | 70.411 | 0.585 | 5.372 | 13.108 |
| Qwen2.5-3B | FP16 SDPA | 62.223 | 1.319 | 4.834 | 12.873 |
| Qwen2.5-3B | FP16 Flash Attention | 56.665 | 1.131 | 4.834 | 11.723 |
| Qwen2.5-3B | FP16 Flash + torch.compile | 47.558 | 0.702 | 4.832 | 9.843 |
| Qwen2.5-3B | FP16 Eager Max Autotune | 33.500 | 0.874 | — | Aggressive Kernel Fusion |
| Gemma-3-4B | FP16 Eager Attention | 100.152 | 2.179 | 8.594 | 11.653 |
| Gemma-3-4B | FP16 SDPA | 93.987 | 1.188 | 8.057 | 11.665 |
| Gemma-3-4B | FP16 Flash Attention | 85.843 | 1.854 | 8.057 | 10.654 |
| Gemma-3-4B | FP16 Flash + torch.compile | 58.833 | 0.814 | 8.053 | 7.306 |
| Gemma-3-4B | FP16 Eager Max Autotune | 41.614 | 0.619 | — | Aggressive Kernel Fusion |
Key Findings¶
- FLOPs Fail as an Energy Proxy: The energy cost per FLOP (\(E/\text{FLOP}\)) varies substantially across components. Attention consistently exhibits the highest \(E/\text{FLOP}\) ratio across all sequence lengths compared to MLP, LM Head, and whole-model aggregates. While dense matrix multiplications in MLP layers efficiently saturate GPU execution units, Attention incurs multi-stage memory traffic, softmax normalization, and irregular access patterns.
- Linear Scaling with Fixed Base Overhead: Component energy scales linearly with operations following \(E(L) \approx E_0 + k \cdot \text{FLOPs}(L)\). As sequence length increases, the fixed overhead \(E_0\) amortizes over more tokens, causing average \(E/\text{FLOP}\) to decay while marginal energy per FLOP (\(\Delta E / \Delta \text{FLOP}\)) remains strictly constant. MLP energy scales linearly with hidden dimension (\(R^2 = 0.995\)), whereas scaling attention heads incurs a heavy constant baseline penalty (\(31.34\,\text{mJ}\) baseline plus \(3.27\,\text{mJ}\) per additional head).
- KV Cache Energy Thresholds: Enabling KV cache during single-token decoding keeps computational load fixed at \(\approx 0.019\,\text{GFLOPs}\) and energy stable (\(\approx 29\text{--}33\,\text{mJ}\)). At 512 input tokens, KV caching yields over \(60\%\) energy reduction for Attention; however, for sequence lengths below 50 tokens, cache maintenance and memory overhead negate benefits, yielding slight negative gains (\(-0.8\%\) to \(-1.7\%\)).
- Architectural Shift Widens Compute vs. Memory Energy Disparity: Moving from Ada Lovelace (RTX 6000) to Hopper (H100) enhances Tensor Core throughput for compute-bound operations, lowering MLP energy per FLOP. Consequently, the \(E/\text{FLOP}\) ratio gap between Attention and MLP expands from \(2.7\times\) on RTX 6000 to \(5.4\times\) on H100, proving that modern GPU enhancements amplify memory-bandwidth bottlenecks in attention.
Highlights & Insights¶
- Pure-Software Repetition Amplification: Eliminates the hardware-access barrier by stretching microsecond execution times into hundreds of milliseconds, enabling standard NVML sensors to capture micro-operator dynamics with \(<9.5\%\) variance.
- Debunking FLOP-Based Green AI Metrics: Empirically demonstrates that substituting FLOPs for actual energy consumption obscures major component-level inefficiencies, pinpointing memory-bound Attention as the primary target for green optimizations.
- Conservation-Based Verification Protocol: Introduces \(\% \text{Capture}\) to audit that component-wise measurements sum up to end-to-end model consumption, consistently verifying \(>90\%\) coverage across architectures.
Limitations & Future Work¶
- FLOP Invisibility Under Aggressive Kernel Fusion: Advanced compiler passes like
torch.compile(mode="max-autotune")fuse matrix multiplications, normalizations, and memory allocations into unified hardware kernels, preventing existing profilers from isolating operator-level FLOP counts. - Hardware Diversity Coverage: While validated across Ada and Hopper architectures, empirical measurements have yet to encompass non-NVIDIA accelerators such as AMD Instinct (ROCm) or specialized inference chips (Google TPU, AWS Inferentia).
- Dynamic Sparsity and MoE Routing Transients: Future extensions should characterize the dynamic energy spikes caused by token load imbalances in MoE routing and dynamic FP4/FP8 quantization regimes.
Related Work & Insights¶
- vs Luccioni et al. (AI Energy Score): AI Energy Score benchmarks end-to-end task-level workloads, obscuring internal variations; CLEAR decomposes models into modular components to identify precise structural energy bottlenecks.
- vs IrEne (Cao et al., 2021): IrEne constructs regression trees on module dimensions but relied on noisy, hardware-instrumented ground truth; CLEAR provides dense, software-accessible, high-precision training targets to enhance predictive modeling.
- vs EDGEPROFILER / FECOM: Prior fine-grained tools targeted TensorFlow runtimes or embedded edge devices; CLEAR natively integrates with PyTorch and targets modern frontier LLM architectures.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ Elegant software amplification bridging the microsecond-to-millisecond hardware sensor gap for internal Transformer primitives.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Profiles 15 models across 4 architectures on 2 GPU generations with full precision, batch size, KV cache, and compilation sweeps.
- Writing Quality: ⭐⭐⭐⭐⭐ Rigorous formalization, clear methodology, and comprehensive energy balance verification.
- Value: ⭐⭐⭐⭐⭐ Foundational for Green AI, energy-aware neural architecture search, and operator-level inference runtime optimization.