Learning Robust Intervention Representations with Delta Embeddings

Conference: ICLR 2026 arXiv: 2508.04492 Code: Project Page Area: Causal Representation Learning / OOD Generalization Keywords: Causal Representation Learning, Delta Embeddings, out-of-distribution, Intervention, Contrastive Learning

TL;DR

This paper proposes the Causal Delta Embedding (CDE) framework, which represents interventions/actions as vector differences between pre- and post-intervention states in the latent space. Three constraints—independence, sparsity, and invariance—are imposed on the delta vectors to learn robust intervention representations. The framework significantly surpasses baselines on the Causal Triplet benchmark in OOD generalization, and autonomously discovers anti-parallel semantic structures for antonymous actions.

Background & Motivation

Understanding how the world responds to actions and interventions is a core capability of AI: Agents operating in dynamic environments must recover the underlying mechanisms that generate and transform data in order to achieve causal reasoning and robust generalization.

Deep learning models fail to generalize under distribution shift: Standard models rely on correlations rather than causal mechanisms, leading to dramatic performance degradation when the data distribution changes (e.g., encountering object–action combinations unseen during training).

Causal representation learning focuses on variable identification but neglects intervention representation: Most CRL work centers on identifying latent causal variables and their relationships (e.g., VAE-based frameworks, score-based methods), while few methods address learning generalizable representations of actions/interventions themselves.

Two key CRL assumptions guide the method design: - Independent Causal Mechanisms (ICM): The data-generating process consists of autonomous and independent modules. - Sparse Mechanism Shift (SMS): A single intervention typically affects only a small number of causal mechanisms.

Two types of OOD generalization challenges: - Compositional shift: Test sets contain object–action combinations unseen during training (e.g., trained on open(door) and close(drawer), tested on open(drawer)). - Systematic shift: Test sets contain entirely novel object categories.

Method

Overall Architecture

The core idea of CDE is to represent an intervention as the vector difference (Delta) between pre- and post-intervention states in the latent space, and to impose three property constraints on this difference vector to produce a "Causal Delta Embedding."

Delta embedding definition: Given an observation pair \((x, \tilde{x})\) before and after an intervention, the delta embedding is defined as \(\delta_a := \phi(\tilde{x}) - \phi(x)\), where \(\phi\) is the encoder.

Under the perfect counterfactual assumption, \(\delta_a = [0 \cdots \tilde{z}_a - z_a \cdots 0]^T\), meaning only the dimensions affected by action \(a\) are non-zero.

Key Designs

1. Three Constraints on Causal Delta Embeddings

• Function: Defines three properties that CDE must satisfy, guiding the design of learning objectives.
• Mechanism: Independence + Sparsity + Invariance → generalizable intervention representations.

• Design Motivation: These three properties derive directly from the ICM and SMS assumptions, ensuring that the learned representations carry causal meaning.

• Independence: The action representation is unaffected by scene attributes and objects that are not acted upon.

• Sparsity: If the SMS assumption holds, \(\delta_a\) should be sparse (most dimensions are zero).
• Invariance: Representations of the same action applied to different objects should be similar (e.g., the representation of "open" should not differ depending on whether the target is a door or a drawer), formalized as \(\text{Var}_{x \sim P(X)}[\delta_a(x)] \approx \mathbf{0}\).

2. Global CDE Model (Model A)

• Function: Learns global-level causal delta embeddings from image pairs.
• Mechanism: ViT-DINO extracts CLS tokens → a causal projector maps them to an \(l\)-dimensional latent space → element-wise subtraction yields the Delta → a classifier predicts the action.
• Design Motivation: The CLS token provides a global image representation, and the subtraction operation naturally satisfies the independence property.

Structural equation modeling: \(\tilde{z}_a = z_a + \delta_a + \epsilon\), where \(\epsilon\) is zero-mean independent noise (corresponding to the actionable counterfactual setting).

3. Patch-wise CDE Model (Model B)

• Function: Handles multi-object scenes where an action affects only a localized region.
• Mechanism: Retains all patch outputs from ViT → computes Delta patch-wise → selects the Top-K patches with the largest L2-norm changes → computes the loss independently for each selected patch.
• Design Motivation: Global embeddings may average out local change signals in complex scenes.

Loss & Training

A weighted combination of three loss functions:

\[\mathcal{L}_{\text{total}} = \mathcal{L}_{\text{CE}} + \alpha_{\text{contrast}} \mathcal{L}_{\text{contrast}} + \alpha_{\text{sparsity}} \mathcal{L}_{\text{sparsity}}\]

1. Cross-entropy loss \(\mathcal{L}_{\text{CE}}\): Ensures that the Delta is useful for action classification.
2. Supervised contrastive loss \(\mathcal{L}_{\text{contrast}}\): Encourages Deltas of the same action class to cluster together and Deltas of different actions to separate (corresponding to the invariance constraint):

\[\mathcal{L}_{\text{contrast}} = \sum_{i=1}^{B} \frac{-1}{|P(i)|} \sum_{p \in P(i)} \log \frac{\exp(\text{sim}(\delta_i, \delta_p)/\tau)}{\sum_{j \neq i} \exp(\text{sim}(\delta_i, \delta_j)/\tau)}\]

1. Sparsity regularization \(\mathcal{L}_{\text{sparsity}} = \frac{1}{B}\sum_i \|\delta_i\|_1\): L1 penalty promotes sparse representations.

Hyperparameters: \(\alpha_{\text{contrast}} = 2.0\), \(\alpha_{\text{sparsity}} = 1.0\), unified across all experiments. The model is trained end-to-end, including encoder updates.

Key Experimental Results

Main Results

Single-object ProcTHOR scenes (synthetic data):

Method	IID Acc.	OOD Comp.	OOD Syst.	Gap↓
Vanilla-R (ResNet)	0.96	0.36	0.48	0.48
Vanilla-V (ViT-DINO)	0.95	0.34	0.47	0.48
ICM-R	0.95	0.41	0.50	0.45
SMS-R	0.96	0.47	0.54	0.42
CDE Global	0.95	significant gain	significant gain	substantially reduced

CDE also demonstrates significant OOD generalization advantages on multi-object and real-world (Epic-Kitchens) scenes.

Ablation Study

Configuration	Effect
All three losses	Best OOD performance
Remove contrastive loss	Reduced invariance, lower OOD accuracy
Remove sparsity regularization	Less compact representations, slight OOD degradation
Cross-entropy only	Degrades to a standard classifier, large OOD drop
Global vs. Patch-wise	Patch-wise superior in multi-object scenes

Key Findings

1. CDE establishes a new state of the art on the Causal Triplet benchmark: It substantially outperforms all baselines on both synthetic and real-world benchmarks.
2. Anti-parallel relationships between antonymous actions are discovered automatically: The delta embeddings of "open" vs. "close" lie in anti-parallel directions in the latent space, entirely without explicit supervision.
3. Independence is naturally satisfied by the Delta computation: No dedicated loss is required; the subtraction operation inherently eliminates scene-level variation.
4. Effectiveness is maintained in the actionable counterfactual setting: Even when \(\epsilon \neq 0\), the classifier remains unaffected (supported by theoretical proof and empirical validation).
5. Sparsity regularization is critical: The L1 penalty ensures that only causally relevant dimensions are activated.

Highlights & Insights

1. Decoupling "learning intervention representations" from "learning variable representations": While most CRL work focuses on recovering causal variables, CDE takes a distinct approach by targeting the representation of interventions/actions themselves—a perspective that is both novel and practically motivated.
2. Minimalist design: Delta = subtraction: Without requiring complex causal discovery or structure learning, causally informative representations are extracted by a simple subtraction of encoder outputs—elegant and effective.
3. Three constraints, three losses: Independence (guaranteed by architectural design) → Sparsity (L1 regularization) → Invariance (contrastive loss); the design rationale is transparent throughout.
4. Spontaneous emergence of anti-parallel semantic structure: The model autonomously learns that "open ↔ close" correspond to opposing directions, providing compelling evidence of causal structure learning.

Limitations & Future Work

1. Requires paired pre- and post-intervention images: In many real-world scenarios, only post-intervention observations are available, making paired data collection infeasible.
2. Limited action vocabulary: The Causal Triplet benchmark contains relatively few action categories (approximately 10+); scalability to large-scale action spaces remains unverified.
3. Restriction to static image pairs: The framework cannot handle sequential actions or continuous changes, and extension to video data is absent.
4. Dependence on the ViT-DINO backbone: Pre-trained visual features provide a strong prior; performance may degrade substantially if the backbone is replaced (e.g., with random initialization).
5. Occlusion and viewpoint variation in real-world scenes: Camera motion and occlusion in Epic-Kitchens may introduce non-causal variation that confounds the delta computation.

Related Work & Insights

• Causal Triplet (Liu et al., 2023): Provides the evaluation framework and SCM model definitions; CDE surpasses all prior methods on this benchmark.
• Von Kügelgen et al. (2021): Theoretically demonstrates that contrastive learning can disentangle causal factors when data augmentations correspond to causal interventions; CDE extends this idea to contrastive learning over intervention pairs.
• DINO (Caron et al., 2021): Self-supervised ViT features provide a strong visual prior for CDE.
• SMS Regularization (Lachapelle et al., 2022): The sparse mechanism shift assumption is concretely instantiated in CDE via L1 regularization.
• Inspiration: The delta embedding paradigm may generalize to broader settings, such as action-effect prediction in reinforcement learning and treatment-effect representation in medical imaging.

Rating

• Novelty: ⭐⭐⭐⭐ — The core idea of Delta = subtraction is concise and powerful, and reorienting CRL from variable identification to intervention representation is a genuinely novel perspective; however, the subtraction operation itself is not technically complex.
• Experimental Thoroughness: ⭐⭐⭐⭐ — The three progressively challenging evaluation settings of Causal Triplet provide comprehensive coverage, and the ablation study is complete; larger-scale evaluation is nonetheless absent.
• Writing Quality: ⭐⭐⭐⭐⭐ — Mathematical definitions are rigorous, the derivation from properties to losses to architecture is logically coherent, and the visualizations (e.g., anti-parallel semantic structure) are highly intuitive.
• Value: ⭐⭐⭐⭐ — Opens a new research direction in causal representation learning (intervention representation) with potential applications in robotics and embodied AI.

Related Papers

• [ICLR 2026] Counterfactual Explanations on Robust Perceptual Geodesics
• [ICLR 2026] Direct Doubly Robust Estimation of Conditional Quantile Contrasts
• [NeurIPS 2025] A Principle of Targeted Intervention for Multi-Agent Reinforcement Learning
• [ICLR 2026] Self-Supervised Learning from Structural Invariance
• [ACL 2026] CausalDetox: Causal Head Selection and Intervention for Language Model Detoxification