LINA: Learning INterventions Adaptively for Physical Alignment and Counterfactual Generation in Diffusion Models

Conference: ECCV 2026
Paper: ECCV Paper
Code: https://opencausalab.github.io/LINA
Area: Image Generation
Keywords: Diffusion Models, Causal Reasoning, Physical Alignment, Counterfactual Generation, Adaptive Intervention

TL;DR

Addressing the physical violations and counterfactual generation failures caused by symmetric denoising in diffusion models, LINA models generation via Causal Scene Graphs, predicts prompt-adaptive token/latent interventions via a lightweight module, and reallocates early denoising computation, achieving SOTA physical alignment without model retraining or online MLLM inference.

Background & Motivation

Diffusion models (DMs) have demonstrated extraordinary generative expressiveness across visual synthesis tasks and are broadly viewed as foundational backbones toward realizing world simulators. However, state-of-the-art visual foundation models consistently falter when confronted with two fundamental tasks: physical alignment (e.g., rendering realistic reflections adhering to optical geometry or respecting density and buoyancy principles) and counterfactual generation (e.g., depicting uncommon or counter-intuitive compositions like "a bird eats a snake" or "a person close to the water and in the sand"). Under standard generation, models routinely render redundant reflections over non-reflective surfaces or collapse counterfactual prompts back into strong statistical priors.

The core tension stems from a fundamental structural flaw in the diffusion denoising paradigm: standard diffusion training forces all spatial image elements to denoise concurrently and symmetrically across timesteps. While the physical world and coherent visual scenes are governed by a strict directional causal hierarchy—flowing from textual specifications \(X\) to directly determined entities \(Y_D\), which subsequently dictate causally downstream physical phenomena \(Y_I\) (such as shadows, reflections, or buoyancy dynamics)—the flattened denoising objective collapses \(X \to Y_D \to Y_I\) into symmetric, parallel correlations (\(X \to Y_D, X \to Y_I\)). Consequently, models fail to enforce directional physical dependencies among generated visual components. Diagnostic analysis further uncovers that representations of texture and physical relations in the prompt embedding space are naturally disentangled, and that the macro visual causal structure is predominantly established during the very initial 2-4 high-noise reverse steps where computational sampling density is disproportionately constrained.

To overcome these structural limitations, generative systems require an intervention framework capable of steering sampling dynamics toward physically coherent causal graphs without destructively fine-tuning foundation weights or incurring prohibitive inference latencies from external multi-modal reasoning loops. The core idea is to formalize image generation through Causal Scene Graphs (CSG), train a lightweight Adaptive Intervention Module (AIM) offline to predict prompt-specific token-level and latent-level intervention strengths, and pair them with a front-loaded causality-aware denoising schedule to achieve training-free, computationally efficient physical alignment and counterfactual generation.

Method

Overall Architecture

LINA operates across two synergistic phases: an offline Phase 1 for Adaptive Intervention Module (AIM) training and an online Phase 2 for causality-guided generation. Grounded in Causal Scene Graphs (CSG) that unify causal dependencies \(E_C\) and spatial layouts \(E_S\), the framework diagnoses physical reasoning failures via the curated Physical Alignment Probe (PAP) dataset. During offline training, hard failure cases from baseline diffusion models are analyzed by an MLLM evaluator via coordinate descent search to extract optimal intervention parameter pairs, which supervise a lightweight T5-MLP regression head. During online inference for any novel prompt, the frozen diffusion model is driven by the predicted token-embedding calibration and latent contrastive guidance along with a causality-aware early-stage computation reallocation, producing physically grounded and counterfactual visual outputs.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input prompt X and unconditional prompt"] --> B["Parser extracts relation tokens & builds neutral prompt"]
    B --> C["Adaptive Intervention Module predicts intervention parameters"]
    C --> D["Causal Scene Graph Dual-Level Intervention<br/>Token calibration + Latent contrastive guidance"]
    D --> E["Causality-Aware Denoising Schedule<br/>Front-loaded high-noise step density reallocation"]
    E --> F["Output physically aligned & counterfactual visual output"]

Key Designs

1. Causal Scene Graph Dual-Level Intervention: Decoupling and selectively amplifying physical relational signals

Standard Classifier-Free Guidance (CFG) cannot distinguish between surface object appearance and directional physical relations. To address this limitation, LINA deploys targeted interventions simultaneously in the prompt embedding space and the visual latent trajectory. In the text embedding space, a lightweight language parser identifies relational tokens corresponding to causal edges (e.g., interaction verbs "hits") and spatial edges (e.g., prepositions "on"), creating a binary mask \(M_{rel}\). The conditioned embedding \(c\) is softly calibrated relative to the unconditional embedding \(u\) using predicted strength \(\gamma_1\):

\[c' = u + \gamma_1 \odot (c - u)\]

In the visual latent space, to isolate the pure causal signal from confounding object textures, the framework automatically generates a neutral reference prompt \(X_{neutral}\) by relaxing relational tokens into generic conjunctions (e.g., replacing "on" with "and"). At timestep \(t\), the standard CFG update is augmented with a contrastive causal guidance term governed by strength \(\gamma_2\):

\[\tilde{\epsilon}_\theta(x_t, t) = \epsilon_\theta(x_t, t, \emptyset) + \gamma_0 (\epsilon_\theta(x_t, t, c') - \epsilon_\theta(x_t, t, \emptyset)) + \gamma_2 (\epsilon_\theta(x_t, t, c') - \epsilon_\theta(x_t, t, c_{neutral}))\]

This dual formulation amplifies directional relational semantics prior to denoising while subtracting generic associative co-occurrence priors in the latent trajectory, steering the model strictly toward the intended causal structure.

2. Adaptive Intervention Module: Offline automated distillation and lightweight online inference

Because prompts vary drastically in causal difficulty, semantic bias, and physical interaction complexity, applying static or heuristic intervention strengths causes severe over-correction or insufficient guidance. LINA resolves the trade-off between adaptive parameterization and inference latency through an automated offline search combined with lightweight online regression.

Offline, researchers build the Physical Alignment Probe (PAP) dataset consisting of 28,700 images across optics, density, and counterfactual generation. An MLLM evaluator (Qwen2.5-VL-72B) extracts bounding boxes and keypoints for rule-based verification, identifying baseline failure subsets \(D_{hard}\). On this hard set, coordinate descent search determines the optimal prompt-specific parameters \((\gamma_1^*, \gamma_2^*)\). An Adaptive Intervention Module (AIM) \(\Phi\), comprising a frozen pre-trained T5 encoder and a lightweight MLP regression head, is trained on \(D_{hard}\) using an \(L_2\) regression loss to predict the normalized parameter ratio:

\[\mathcal{L}_{reg} = \sum_{i=1}^N \left\| \Phi(X_i) - (\gamma_{1, i}^*, \gamma_{2, i}^* / \gamma_0) \right\|_2^2\]

During online generation, AIM evaluates any incoming prompt instantaneously, bypassing the excessive compute, multi-turn latency, and context degradation associated with runtime MLLM corrector loops.

3. Causality-Aware Denoising Schedule: Reallocating computational density to the structural formation window

Empirical probing of intermediate diffusion latent trajectories reveals that the macro causal structure of a scene is resolved almost entirely in the earliest high-noise stages—with 97.8% of successful causal topologies established within the initial 2-4 reverse iterations (e.g., steps 26-24 of a 28-step schedule). Uniform or standard sampling schedules allocate the vast majority of computation to downstream high-frequency texture refinement, leaving the delicate causal emergence phase computationally constrained.

To address this temporal mismatch, LINA implements a computation reallocation strategy built on nonlinear time-shifting:

\[\tau_s = \frac{s \cdot \tau}{1 + (s - 1) \cdot \tau}\]

By increasing the schedule shift parameter \(s\) (e.g., elevating \(s\) from the official default of 1.15 in SD-3.5-large to 1.5), the framework compresses step lengths during early high-noise stages while maintaining the overall inference budget. This front-loads sampling resolution directly into the critical window where causal scene topologies emerge, ensuring that the dual-level intervention mechanisms operate with maximum efficacy before visual layouts lock in.

A Worked Example

Consider generating an image with the counterfactual optical prompt "A blue cube on a small mirror" (where the mirror's area is significantly smaller than the supporting base): 1. Parsing & Neutral Prompt Generation: The syntactic parser detects the spatial preposition "on", creating relation mask \(M_{rel}\), and derives the neutral reference prompt \(X_{neutral}\): "A blue cube and a small mirror". 2. Adaptive Strength Prediction: The prompt is processed by AIM's T5-MLP head, predicting calibrated scale \(\hat{\gamma}_1 = 1.35\) and contrastive ratio \(\hat{\gamma}_2 / \gamma_0 = 0.42\) tuned to counterfactual layout resistance. 3. Causality-Aware Time Discretization: Time-shift parameter \(s\) is set to 1.5, heavily concentrating discrete sampling steps into the initial structural crystallization interval. 4. Dual Guided Sampling: Denoising proceeds with calibrated embedding \(c'\) steering relational edge strength, while the contrastive latent subtraction \((\epsilon_\theta(c') - \epsilon_\theta(c_{neutral}))\) actively penalizes spurious mirror reflections generated outside the physical boundary of the mirror. 5. Final Output: The resulting generation confines the cube's reflection strictly within the mirror's geometric boundary, leaving surrounding surface regions entirely matte and physically coherent.

Key Experimental Results

Main Results

The main benchmark evaluates physical alignment across Optics and Density subsets alongside counterfactual simulation on the Winoground benchmark (171 unambiguous pairs) and the PAP-OOD test set. Success rates (% ↑) are determined via deterministic geometric rule verification over Qwen2.5-VL-extracted primitives.

Method	Backbone	Optics (%) ↑	Density (%) ↑	Winoground (%) ↑	PAP-OOD (%) ↑
SD-3.5 (Baseline)	SD-3.5-large	80.4 ± 4.0	54.2 ± 4.5	54.4 ± 3.4	69.3 ± 3.8
FLUX.1 (Baseline)	FLUX.1-Krea-dev	86.9 ± 3.3	64.3 ± 4.1	65.5 ± 4.2	80.6 ± 3.2
SD-3.5 + LMD (Layout Guidance)	SD-3.5-large	80.5 ± 3.7	81.5 ± 3.4	73.1 ± 4.0	75.2 ± 3.5
SD-3.5 + PPAD (MLLM-in-the-loop)	SD-3.5-large	91.7 ± 2.6	76.2 ± 3.3	62.6 ± 4.3	74.1 ± 3.4
SD-3.5 + LoRA (Finetuning)	SD-3.5-large	95.9 ± 4.0	91.3 ± 3.9	57.3 ± 4.5	72.0 ± 3.8
Lina (SD-3.5, Ours)	SD-3.5-large	97.4 ± 2.1	92.3 ± 2.0	79.5 ± 2.0	84.3 ± 2.4
Lina (FLUX.1, Ours)	FLUX.1-Krea-dev	96.8 ± 1.9	94.0 ± 2.1	83.0 ± 2.5	86.1 ± 2.2

Ablation Study

Ablation experiments conducted on SD-3.5-large isolate the contribution of each algorithmic component across PAP-Optics, PAP-Density, and Winoground.

Configuration	Optics (%) ↑	Density (%) ↑	Winoground (%) ↑	Note
Lina (Full Model)	97.4 ± 2.1	92.3 ± 2.0	79.5 ± 2.0	Full dual-level interventions + adaptive schedule
w/o \(\gamma_1\) (no token calibration)	85.1 ± 3.6	80.5 ± 3.8	60.2 ± 3.5	Winoground plummets by 19.3%, breaking counterfactual reasoning
w/o \(\gamma_2\) (no latent contrastive guidance)	81.3 ± 3.9	78.0 ± 4.1	74.9 ± 3.4	Physical optics alignment degrades heavily toward baseline
Fixed \(\gamma\) (static mean intervention)	90.5 ± 3.1	85.2 ± 3.3	68.4 ± 3.0	Demonstrates critical need for prompt-specific adaptive values
Std. Schedule (uniform time-shift)	92.3 ± 2.8	88.1 ± 3.0	74.3 ± 2.7	Confirms importance of front-loaded computation reallocation
Baseline SD-3.5	80.4 ± 4.0	54.2 ± 4.5	54.4 ± 3.4	Unmodified base model with no interventions

Key Findings

• Complementary Dual Interventions: Token calibration (\(\gamma_1\)) governs semantic bias override (removing it drops Winoground by 19.3%), whereas latent contrastive guidance (\(\gamma_2\)) isolates spatial physical laws (removing it drops Optics by 16.1%), proving that language-space calibration and latent dynamics steering address distinct failure modes.
• Superiority Over Parameter Tuning and Runtime Loops: LoRA fine-tuning exhibits catastrophic overfitting to correlated priors (scoring only 57.3% on Winoground), while runtime MLLM correctors (PPAD) suffer from the "identification-correction gap" alongside +372% computational overhead. In contrast, LINA achieves superior accuracy with only +25% computational overhead.
• Temporal Generalization to Video Foundation Models: Extending LINA directly to the SOTA video diffusion model Wan2.2-T2V-A14B elevates physical consistency in dynamic sequences (e.g., fluid floating vs. sinking) from a baseline of 29.5% to 58.0%, demonstrating seamless transfer across generative modalities.

Highlights & Insights

• Root Diagnosis of Generative Physical Failure: The paper formally demonstrates that the persistent physical inaccuracies of modern diffusion models originate from symmetric parallel denoising that flattens directional causal hierarchies (\(X \to Y_D \to Y_I\)), providing a rigorous theoretical framework for future generative alignment research.
• Efficient Inference Without Weight Modification: By restricting intervention to prompt embedding calibration and contrastive latent guidance parameterized by a lightweight offline-trained module, LINA avoids expensive foundation model retraining while requiring zero multi-modal API calls during inference.
• Empirical Discovery of Causal Emergence Timings: Pinpointing that 97.8% of causal scene topologies are frozen within the initial 2-4 reverse denoising steps provides a principled foundation for non-uniform sampling schedules and causal computation reallocation in visual generative models.

Limitations & Future Work

• Scope of Physical Domains: Evaluation and benchmark probing currently focus primarily on optical reflection, fluid density/buoyancy, and basic counterfactual containment; more complex thermodynamic, non-rigid, or topological transformations remain unexplored.
• Dependence on Rule-Based Parsing: Neutral reference prompt synthesis and relation token masking currently rely on discrete linguistic parsing tools (spaCy), which may encounter brittle token assignments when handling complex, non-standard syntactic inputs.
• Future Directions: Developing end-to-end continuous relational extractors directly within diffusion text encoders, and extending adaptive causal interventions to interactive 3D scene generation and autonomous robotics world models.

Related Work & Insights

• vs. Prompt Engineering & Layout Guidance (e.g., LMD): Layout-based methods rely on LLMs to generate bounding boxes to steer spatial cross-attention. While competent for object co-location, they fail to enforce implicit physical laws (such as reflections or gravity interactions). LINA directly steers generation dynamics to respect physical dependencies.
• vs. MLLM-in-the-loop Correction (e.g., PPAD): Iterative refinement methods insert external vision-language models into the denoising loop. They incur massive computational overhead (+372%) and remain constrained by the "identification-correction gap" (models spot errors but cannot guide diffusion latents out of bad attractors). LINA moves all MLLM guidance offline, adding only +25% compute at inference.
• vs. Parameter Finetuning & Adapters (e.g., LoRA): Supervised fine-tuning of diffusion weights biases the network toward training distribution correlations, degrading performance on out-of-distribution counterfactual tasks. LINA preserves all pre-trained weights, tuning only sampling dynamics adaptively to retain broad generalization.

Rating

• Novelty: ⭐⭐⭐⭐⭐ [Pioneering integration of Causal Scene Graphs with diffusion sampling dynamics, introducing dual-level interventions and causal schedule reallocation]
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Extensive validation across 28.7k-image PAP corpus, multi-backbone image and video benchmarks, comprehensive ablations, and rigorous inter-evaluator agreement checks]
• Writing Quality: ⭐⭐⭐⭐⭐ [Clear mathematical problem formulation, disciplined structure, logical transitions, and robust empirical evidence]
• Value: ⭐⭐⭐⭐⭐ [Provides an efficient, training-free blueprint for physical consistency and counterfactual reasoning essential for generative world simulators]