Skip to content

Dissecting Transformers: A CLEAR Perspective Towards Green AI

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/divyanash911/CLEAR
Area: Others
Keywords: Green AI, Energy Efficiency, Fine-Grained Profiling, Transformer Architecture, NVML

TL;DR

Addressing the temporal mismatch between microsecond-scale operator execution and millisecond-scale power sensors, this paper proposes CLEAR, a software-based fine-grained energy measurement pipeline that profiles individual Transformer components across 15 models, revealing that the Attention block consumes significantly more energy per FLOP than the overall model.

Background & Motivation

The ubiquitous adoption of Large Language Models (LLMs) has raised urgent environmental concerns regarding carbon emissions and grid consumption. ChatGPT-4o alone is estimated to emit around 150,000 metric tons of CO2e in 2025, matching the annual carbon footprint of 30,000 gasoline-powered passenger vehicles. Crucially, whereas training is an episodic, one-off investment, inference operates continuously at immense global scale, handling hundreds of millions of user queries daily across systems like Gemini, Claude, and ChatGPT. Consequently, even fractional reductions in per-inference energy translate into substantial global savings, making inference the paramount target for Green AI research.

Despite this imperative, prevailing sustainability evaluations remain limited to coarse, model-level or dataset-level aggregations. Prior benchmarks typically measure cumulative energy over long evaluation workloads, reporting aggregate Joules and deriving coarse average figures. Such macroscopic tracking treats LLMs as monolithic black boxes, obscuring the disproportionate energy consumption of internal modules like Attention, MLP, LayerNorm, and the LM Head. Meanwhile, hardware-centric profiling approaches require intrusive, expensive tracing instruments that cannot easily scale across diverse computing clusters.

The fundamental technical obstacle to fine-grained profiling is the severe temporal granularity mismatch between GPU kernel execution and onboard power sensors. Individual Transformer sub-operations (such as QKV projections, attention heads, or normalization passes) execute within 10 to 100 microseconds, whereas standard hardware telemetry interfaces (such as NVIDIA NVML) sample power only every 20 to 50 milliseconds. A single operator execution falling entirely between two sensor samples records an erroneous reading of zero millijoules; conversely, measurements spanning an arbitrary window are overwhelmed by background idle power noise. The core idea is to establish CLEAR (Component-Level Energy Assessment via Repetitions), a software-only profiling pipeline that caches forward activations and executes thousands of unpadded repetitions to bridge the temporal gap, verified by a closed-loop validation of energy completeness and cross-trial consistency.

Method

Overall Architecture

CLEAR provides a modular, lightweight, and hardware-agnostic three-stage pipeline implemented entirely at the application layer. It systematically dissects Transformer architectures into their constituent computational primitives: Attention blocks, feed-forward Multi-Layer Perceptrons (MLPs), Normalization layers, input Embeddings, and final Language Modeling (LM) Heads. The pipeline comprises: (1) Activation Store, which intercepts and caches genuine layer inputs; (2) Amplification Strategy, which scales execution duration to overwhelm telemetry noise; and (3) Validation, which assesses the completeness and consistency of the measured energy ledger.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input Sequence & Model"] --> B["Stage 1: Activation Store<br/>Intercept and cache real activation tensors via hooks"]
    B --> C["Stage 2: Amplification Strategy<br/>Execute N back-to-back repetitions to span NVML windows"]
    C --> D["Stage 3: Validation Protocol<br/>Verify completeness ratio and cross-trial variance"]
    D --> E["Component-Level Energy Profile<br/>Attention / MLP / Norm / LM Head breakdown"]

Key Designs

1. Activation Store: Decoupling Component Execution via Cached Forward Tensors

Measuring components in complete isolation without corrupting input statistics requires preserving the exact operational distribution across layers. CLEAR inserts non-intrusive forward hooks across the computation graph for every primary component \(c \in \mathcal{A} = \{\text{attn}, \text{mlp}, \text{norm}, \text{embedding}, \text{lm\_head}\}\). During an initial forward pass, the hooks extract and cache the input activation tensors \(a_c \in \mathbb{R}^d\) for all sequence positions. This cache enables subsequent isolated replays of each sub-layer under true representative workloads, avoiding artificial synthetic tensor artifacts and decoupling target operators from downstream network latency and synchronization overheads.

2. Amplification Strategy: Rapid Consecutive Repetition to Eliminate Sensor Mismatch

Because individual component kernels run in roughly 10โ€“100 \(\mu\)s, single-invocation readings against NVML's 20โ€“50 ms sampling rate fail catastrophically. CLEAR solves this by chaining \(N\) uninterrupted executions of component \(c\) directly on cached tensor \(a_c\). By eliminating CUDA kernel gaps, the aggregated execution duration extends to hundreds of milliseconds. The total energy \(E_{\text{tot}}^c\) scales up proportionally while baseline idle power noise \(\varepsilon\) remains constant, effectively shrinking the per-execution noise term \(\varepsilon/N\) to near-zero: $$ \hat{E}c = \frac{E $$ The repetition budget is calibrated to }} - E_{\text{start}}}{N} = \frac{E_{\text{tot}}^c \pm \varepsilon}{N\(N = 10{,}000\) for small microsecond components and \(N = 1{,}000\) for full-model forwards. Each measurement is conducted across \(T = 20\) independent trials with short pauses between runs to allow sensor resets, reporting mean values and standard deviations to guarantee numerical precision.

3. Validation Protocol: Verification via Completeness and Consistency Metrics

Given the absence of prior ground-truth data for micro-component energy attribution, CLEAR establishes a rigorous validation protocol along two complementary axes. First, consistency requires low variance across repeated runs: for components consuming over 5 mJ, the standard deviation is verified to stay below 9.5% of the mean, narrowing to around 1% for larger workloads. Second, completeness assesses whether the summation of individual components accounts for the empirical end-to-end model consumption: $$ \text{\%Capture} = \frac{\sum_{i=1}^M \bar{E}i}{\bar{E} \times 100\% $$ Across diverse architectures, CLEAR consistently achieves 90% to 102% capture rates at both the block and model levels, mathematically confirming that all non-negligible energy-consuming primitives are reliably accounted for.}}

Key Experimental Results

Main Results

Experiments were conducted primarily on NVIDIA Ada Lovelace GPUs (RTX 6000 Ada and RTX 5000 Ada), with architectural generalizability validated on NVIDIA Hopper H100 GPUs across 15 models spanning Encoder-only, Decoder-only, Seq2Seq, and Mixture-of-Experts (MoE) families.

The following table presents the component-level energy breakdown and capture rates for the open-source MoE model GPT-OSS-20B across varying sequence lengths:

Component / Metric 8 Tokens 32 Tokens 64 Tokens 96 Tokens 128 Tokens
Attention Block (mJ) 53.261 \(\pm\) 1.677 64.147 \(\pm\) 0.686 75.161 \(\pm\) 0.760 93.910 \(\pm\) 0.779 100.701 \(\pm\) 1.045
MLP Block (mJ) 685.408 \(\pm\) 12.61 776.905 \(\pm\) 3.166 867.687 \(\pm\) 0.867 958.134 \(\pm\) 1.406 1046.200 \(\pm\) 1.187
Norm. All (mJ) 9.324 \(\pm\) 0.729 10.787 \(\pm\) 0.825 12.702 \(\pm\) 1.056 13.443 \(\pm\) 1.422 14.639 \(\pm\) 1.108
LM Head (mJ) 443.391 \(\pm\) 1.108 452.139 \(\pm\) 0.988 460.383 \(\pm\) 0.988 475.220 \(\pm\) 1.265 483.515 \(\pm\) 1.242
Block Measured (mJ) 731.905 \(\pm\) 12.456 856.309 \(\pm\) 1.428 951.869 \(\pm\) 0.805 1057.010 \(\pm\) 0.881 1157.197 \(\pm\) 1.181
Block Capture Ratio (%Capture) 102.20% 99.48% 100.39% 100.80% 100.38%
Model Measured (mJ) 18447.50 \(\pm\) 63.78 21366.69 \(\pm\) 103.48 24126.47 \(\pm\) 12.67 26634.05 \(\pm\) 15.33 28801.98 \(\pm\) 2.87
Model Capture Ratio (%Capture) 97.65% 98.33% 96.63% 97.06% 98.13%

Ablation Study

The ablation analysis investigates the impact of self-attention operator implementations, compiler optimizations, and sequence length on energy consumption and compute efficiency (mJ/GFLOP). The table below details Qwen2.5-3B and Gemma-3-4B at 256 input tokens:

Model & Configuration Energy (mJ) Std. Dev. (mJ) FLOPs (GFLOPs) Energy per GFLOP (mJ/GFLOP)
Qwen2.5-3B (256 Tokens)
Eager Attention (Baseline) 70.411 0.585 5.372 13.108
SDPA 62.223 1.319 4.834 12.873
Flash Attention 56.665 1.131 4.834 11.723
Eager w/ torch.compile() 59.949 0.818 5.369 11.166
Flash w/ torch.compile() 47.558 0.702 4.832 9.843
Eager Max Autotune 33.500 0.874 โ€“ โ€“
Eager Reduce Overhead 36.626 1.095 โ€“ โ€“
Gemma-3-4B (256 Tokens)
Eager Attention (Baseline) 100.152 2.179 8.594 11.653
SDPA 93.987 1.188 8.057 11.665
Flash Attention 85.843 1.854 8.057 10.654
Eager w/ torch.compile() 60.907 0.696 8.590 7.091
Flash w/ torch.compile() 58.833 0.814 8.053 7.306
Eager Max Autotune 41.614 0.619 โ€“ โ€“
Eager Reduce Overhead 44.674 0.161 โ€“ โ€“

Key Findings

  • FLOPs Fail as a Direct Energy Proxy: The Attention mechanism consistently exhibits a much higher Energy/FLOP ratio than MLP, LM Head, and the aggregate model across all sequence lengths. Because MLPs comprise dense matrix multiplications that fully saturate Tensor Cores, they process more compute per unit energy. In contrast, Attention incurs substantial memory movement and synchronization overhead from Softmax and tiled memory fetches.
  • Decomposition of Energy Scaling: Component energy scales linearly according to \(E(L) \approx E_0 + k \cdot \text{FLOPs}(L)\). While the fixed overhead \(E_0\) amortizes as context grows (reducing average mJ/GFLOP), the marginal cost per FLOP (\(k\)) remains constant, with Attention displaying a much steeper marginal slope \(k\) than MLP.
  • KV Cache Amortization vs. Short-Sequence Overhead: In autoregressive generation, KV caching bounds per-step Attention FLOPs to roughly 0.019 GFLOPs, reducing energy by over 60% at 512 tokens. However, for context lengths below 50 tokens, cache maintenance introduces minor negative savings.
  • Kernel Fusion Dominates Algorithm Upgrades: Aggressive compiler fusion (torch.compile under Max Autotune and Reduce Overhead) slashes energy consumption by over 50%, outperforming the transition from Eager Attention to Flash Attention alone.

Highlights & Insights

  • Implements an elegant, software-only amplification paradigm that circumvents hardware sensor sampling limitations without requiring physical probes or invasive instrumentation.
  • Validates the energy decomposition of deep networks through rigorous completeness (%Capture \(\ge 90\%\)) and consistency metrics, confirming the near-total capture of computational dissipation.
  • Establishes that scaling hidden dimensions expands energy linearly in MLPs (\(R^2=0.998\)), whereas scaling attention heads introduces both non-trivial static overheads and amplified marginal per-head costs.

Limitations & Future Work

  • The PyTorch Profiler is unable to extract FLOP counts when aggressive kernel fusion merges operators into monolithic hardware routines (e.g., in Max Autotune).
  • Empirical evaluations focus primarily on NVIDIA microarchitectures (Ada Lovelace and Hopper), leaving AMD CDNA, Apple Silicon, and dedicated accelerators for future verification.
  • The profiling framework has not yet been extended to dynamic speculative decoding pipelines or structured MoE routing load imbalances under diverse sampling distributions.
  • vs IrEne (Cao et al., 2021): IrEne utilizes regression-based models over architectural parameters but relies on specialized hardware counters; CLEAR provides an accessible, software-driven foundation that supplies dense empirical ground truths.
  • vs AI Energy Score (Luccioni et al., 2025): AI Energy Score benchmarks models at a holistic task level; CLEAR provides the microscopic causal explanation for why certain architectural choices incur higher environmental footprints.

Rating

  • Novelty: โญโญโญโญโญ Pioneering software-based amplification resolving the physical microsecond-to-millisecond telemetry mismatch.
  • Experimental Thoroughness: โญโญโญโญโญ Evaluates 15 models across 4 architectural paradigms, multiple GPU generations, and extensive optimization ablations.
  • Writing Quality: โญโญโญโญโญ Exceptional narrative flow, mathematically sound framing, and clear visualization of fine-grained dynamics.
  • Value: โญโญโญโญโญ Foundational tool and empirical reference for architectural design decisions in Green AI.