Dissecting Transformers: A CLEAR Perspective Towards Green AI¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/divyanash911/CLEAR
Area: Others
Keywords: Green AI, Energy Efficiency, Fine-Grained Profiling, Transformer Architecture, NVML
TL;DR¶
Addressing the temporal mismatch between microsecond-scale operator execution and millisecond-scale power sensors, this paper proposes CLEAR, a software-based fine-grained energy measurement pipeline that profiles individual Transformer components across 15 models, revealing that the Attention block consumes significantly more energy per FLOP than the overall model.
Background & Motivation¶
The ubiquitous adoption of Large Language Models (LLMs) has raised urgent environmental concerns regarding carbon emissions and grid consumption. ChatGPT-4o alone is estimated to emit around 150,000 metric tons of CO2e in 2025, matching the annual carbon footprint of 30,000 gasoline-powered passenger vehicles. Crucially, whereas training is an episodic, one-off investment, inference operates continuously at immense global scale, handling hundreds of millions of user queries daily across systems like Gemini, Claude, and ChatGPT. Consequently, even fractional reductions in per-inference energy translate into substantial global savings, making inference the paramount target for Green AI research.
Despite this imperative, prevailing sustainability evaluations remain limited to coarse, model-level or dataset-level aggregations. Prior benchmarks typically measure cumulative energy over long evaluation workloads, reporting aggregate Joules and deriving coarse average figures. Such macroscopic tracking treats LLMs as monolithic black boxes, obscuring the disproportionate energy consumption of internal modules like Attention, MLP, LayerNorm, and the LM Head. Meanwhile, hardware-centric profiling approaches require intrusive, expensive tracing instruments that cannot easily scale across diverse computing clusters.
The fundamental technical obstacle to fine-grained profiling is the severe temporal granularity mismatch between GPU kernel execution and onboard power sensors. Individual Transformer sub-operations (such as QKV projections, attention heads, or normalization passes) execute within 10 to 100 microseconds, whereas standard hardware telemetry interfaces (such as NVIDIA NVML) sample power only every 20 to 50 milliseconds. A single operator execution falling entirely between two sensor samples records an erroneous reading of zero millijoules; conversely, measurements spanning an arbitrary window are overwhelmed by background idle power noise. The core idea is to establish CLEAR (Component-Level Energy Assessment via Repetitions), a software-only profiling pipeline that caches forward activations and executes thousands of unpadded repetitions to bridge the temporal gap, verified by a closed-loop validation of energy completeness and cross-trial consistency.
Method¶
Overall Architecture¶
CLEAR provides a modular, lightweight, and hardware-agnostic three-stage pipeline implemented entirely at the application layer. It systematically dissects Transformer architectures into their constituent computational primitives: Attention blocks, feed-forward Multi-Layer Perceptrons (MLPs), Normalization layers, input Embeddings, and final Language Modeling (LM) Heads. The pipeline comprises: (1) Activation Store, which intercepts and caches genuine layer inputs; (2) Amplification Strategy, which scales execution duration to overwhelm telemetry noise; and (3) Validation, which assesses the completeness and consistency of the measured energy ledger.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Input Sequence & Model"] --> B["Stage 1: Activation Store<br/>Intercept and cache real activation tensors via hooks"]
B --> C["Stage 2: Amplification Strategy<br/>Execute N back-to-back repetitions to span NVML windows"]
C --> D["Stage 3: Validation Protocol<br/>Verify completeness ratio and cross-trial variance"]
D --> E["Component-Level Energy Profile<br/>Attention / MLP / Norm / LM Head breakdown"]
Key Designs¶
1. Activation Store: Decoupling Component Execution via Cached Forward Tensors
Measuring components in complete isolation without corrupting input statistics requires preserving the exact operational distribution across layers. CLEAR inserts non-intrusive forward hooks across the computation graph for every primary component \(c \in \mathcal{A} = \{\text{attn}, \text{mlp}, \text{norm}, \text{embedding}, \text{lm\_head}\}\). During an initial forward pass, the hooks extract and cache the input activation tensors \(a_c \in \mathbb{R}^d\) for all sequence positions. This cache enables subsequent isolated replays of each sub-layer under true representative workloads, avoiding artificial synthetic tensor artifacts and decoupling target operators from downstream network latency and synchronization overheads.
2. Amplification Strategy: Rapid Consecutive Repetition to Eliminate Sensor Mismatch
Because individual component kernels run in roughly 10โ100 \(\mu\)s, single-invocation readings against NVML's 20โ50 ms sampling rate fail catastrophically. CLEAR solves this by chaining \(N\) uninterrupted executions of component \(c\) directly on cached tensor \(a_c\). By eliminating CUDA kernel gaps, the aggregated execution duration extends to hundreds of milliseconds. The total energy \(E_{\text{tot}}^c\) scales up proportionally while baseline idle power noise \(\varepsilon\) remains constant, effectively shrinking the per-execution noise term \(\varepsilon/N\) to near-zero: $$ \hat{E}c = \frac{E $$ The repetition budget is calibrated to }} - E_{\text{start}}}{N} = \frac{E_{\text{tot}}^c \pm \varepsilon}{N\(N = 10{,}000\) for small microsecond components and \(N = 1{,}000\) for full-model forwards. Each measurement is conducted across \(T = 20\) independent trials with short pauses between runs to allow sensor resets, reporting mean values and standard deviations to guarantee numerical precision.
3. Validation Protocol: Verification via Completeness and Consistency Metrics
Given the absence of prior ground-truth data for micro-component energy attribution, CLEAR establishes a rigorous validation protocol along two complementary axes. First, consistency requires low variance across repeated runs: for components consuming over 5 mJ, the standard deviation is verified to stay below 9.5% of the mean, narrowing to around 1% for larger workloads. Second, completeness assesses whether the summation of individual components accounts for the empirical end-to-end model consumption: $$ \text{\%Capture} = \frac{\sum_{i=1}^M \bar{E}i}{\bar{E} \times 100\% $$ Across diverse architectures, CLEAR consistently achieves 90% to 102% capture rates at both the block and model levels, mathematically confirming that all non-negligible energy-consuming primitives are reliably accounted for.}}
Key Experimental Results¶
Main Results¶
Experiments were conducted primarily on NVIDIA Ada Lovelace GPUs (RTX 6000 Ada and RTX 5000 Ada), with architectural generalizability validated on NVIDIA Hopper H100 GPUs across 15 models spanning Encoder-only, Decoder-only, Seq2Seq, and Mixture-of-Experts (MoE) families.
The following table presents the component-level energy breakdown and capture rates for the open-source MoE model GPT-OSS-20B across varying sequence lengths:
| Component / Metric | 8 Tokens | 32 Tokens | 64 Tokens | 96 Tokens | 128 Tokens |
|---|---|---|---|---|---|
| Attention Block (mJ) | 53.261 \(\pm\) 1.677 | 64.147 \(\pm\) 0.686 | 75.161 \(\pm\) 0.760 | 93.910 \(\pm\) 0.779 | 100.701 \(\pm\) 1.045 |
| MLP Block (mJ) | 685.408 \(\pm\) 12.61 | 776.905 \(\pm\) 3.166 | 867.687 \(\pm\) 0.867 | 958.134 \(\pm\) 1.406 | 1046.200 \(\pm\) 1.187 |
| Norm. All (mJ) | 9.324 \(\pm\) 0.729 | 10.787 \(\pm\) 0.825 | 12.702 \(\pm\) 1.056 | 13.443 \(\pm\) 1.422 | 14.639 \(\pm\) 1.108 |
| LM Head (mJ) | 443.391 \(\pm\) 1.108 | 452.139 \(\pm\) 0.988 | 460.383 \(\pm\) 0.988 | 475.220 \(\pm\) 1.265 | 483.515 \(\pm\) 1.242 |
| Block Measured (mJ) | 731.905 \(\pm\) 12.456 | 856.309 \(\pm\) 1.428 | 951.869 \(\pm\) 0.805 | 1057.010 \(\pm\) 0.881 | 1157.197 \(\pm\) 1.181 |
| Block Capture Ratio (%Capture) | 102.20% | 99.48% | 100.39% | 100.80% | 100.38% |
| Model Measured (mJ) | 18447.50 \(\pm\) 63.78 | 21366.69 \(\pm\) 103.48 | 24126.47 \(\pm\) 12.67 | 26634.05 \(\pm\) 15.33 | 28801.98 \(\pm\) 2.87 |
| Model Capture Ratio (%Capture) | 97.65% | 98.33% | 96.63% | 97.06% | 98.13% |
Ablation Study¶
The ablation analysis investigates the impact of self-attention operator implementations, compiler optimizations, and sequence length on energy consumption and compute efficiency (mJ/GFLOP). The table below details Qwen2.5-3B and Gemma-3-4B at 256 input tokens:
| Model & Configuration | Energy (mJ) | Std. Dev. (mJ) | FLOPs (GFLOPs) | Energy per GFLOP (mJ/GFLOP) |
|---|---|---|---|---|
| Qwen2.5-3B (256 Tokens) | ||||
| Eager Attention (Baseline) | 70.411 | 0.585 | 5.372 | 13.108 |
| SDPA | 62.223 | 1.319 | 4.834 | 12.873 |
| Flash Attention | 56.665 | 1.131 | 4.834 | 11.723 |
| Eager w/ torch.compile() | 59.949 | 0.818 | 5.369 | 11.166 |
| Flash w/ torch.compile() | 47.558 | 0.702 | 4.832 | 9.843 |
| Eager Max Autotune | 33.500 | 0.874 | โ | โ |
| Eager Reduce Overhead | 36.626 | 1.095 | โ | โ |
| Gemma-3-4B (256 Tokens) | ||||
| Eager Attention (Baseline) | 100.152 | 2.179 | 8.594 | 11.653 |
| SDPA | 93.987 | 1.188 | 8.057 | 11.665 |
| Flash Attention | 85.843 | 1.854 | 8.057 | 10.654 |
| Eager w/ torch.compile() | 60.907 | 0.696 | 8.590 | 7.091 |
| Flash w/ torch.compile() | 58.833 | 0.814 | 8.053 | 7.306 |
| Eager Max Autotune | 41.614 | 0.619 | โ | โ |
| Eager Reduce Overhead | 44.674 | 0.161 | โ | โ |
Key Findings¶
- FLOPs Fail as a Direct Energy Proxy: The Attention mechanism consistently exhibits a much higher Energy/FLOP ratio than MLP, LM Head, and the aggregate model across all sequence lengths. Because MLPs comprise dense matrix multiplications that fully saturate Tensor Cores, they process more compute per unit energy. In contrast, Attention incurs substantial memory movement and synchronization overhead from Softmax and tiled memory fetches.
- Decomposition of Energy Scaling: Component energy scales linearly according to \(E(L) \approx E_0 + k \cdot \text{FLOPs}(L)\). While the fixed overhead \(E_0\) amortizes as context grows (reducing average mJ/GFLOP), the marginal cost per FLOP (\(k\)) remains constant, with Attention displaying a much steeper marginal slope \(k\) than MLP.
- KV Cache Amortization vs. Short-Sequence Overhead: In autoregressive generation, KV caching bounds per-step Attention FLOPs to roughly 0.019 GFLOPs, reducing energy by over 60% at 512 tokens. However, for context lengths below 50 tokens, cache maintenance introduces minor negative savings.
- Kernel Fusion Dominates Algorithm Upgrades: Aggressive compiler fusion (
torch.compileunder Max Autotune and Reduce Overhead) slashes energy consumption by over 50%, outperforming the transition from Eager Attention to Flash Attention alone.
Highlights & Insights¶
- Implements an elegant, software-only amplification paradigm that circumvents hardware sensor sampling limitations without requiring physical probes or invasive instrumentation.
- Validates the energy decomposition of deep networks through rigorous completeness (%Capture \(\ge 90\%\)) and consistency metrics, confirming the near-total capture of computational dissipation.
- Establishes that scaling hidden dimensions expands energy linearly in MLPs (\(R^2=0.998\)), whereas scaling attention heads introduces both non-trivial static overheads and amplified marginal per-head costs.
Limitations & Future Work¶
- The PyTorch Profiler is unable to extract FLOP counts when aggressive kernel fusion merges operators into monolithic hardware routines (e.g., in Max Autotune).
- Empirical evaluations focus primarily on NVIDIA microarchitectures (Ada Lovelace and Hopper), leaving AMD CDNA, Apple Silicon, and dedicated accelerators for future verification.
- The profiling framework has not yet been extended to dynamic speculative decoding pipelines or structured MoE routing load imbalances under diverse sampling distributions.
Related Work & Insights¶
- vs IrEne (Cao et al., 2021): IrEne utilizes regression-based models over architectural parameters but relies on specialized hardware counters; CLEAR provides an accessible, software-driven foundation that supplies dense empirical ground truths.
- vs AI Energy Score (Luccioni et al., 2025): AI Energy Score benchmarks models at a holistic task level; CLEAR provides the microscopic causal explanation for why certain architectural choices incur higher environmental footprints.
Rating¶
- Novelty: โญโญโญโญโญ Pioneering software-based amplification resolving the physical microsecond-to-millisecond telemetry mismatch.
- Experimental Thoroughness: โญโญโญโญโญ Evaluates 15 models across 4 architectural paradigms, multiple GPU generations, and extensive optimization ablations.
- Writing Quality: โญโญโญโญโญ Exceptional narrative flow, mathematically sound framing, and clear visualization of fine-grained dynamics.
- Value: โญโญโญโญโญ Foundational tool and empirical reference for architectural design decisions in Green AI.