Domain Generalization via Text-Anchored Information Bottleneck

Conference: ECCV 2026
Paper: ECCV Official
Area: Multimodal VLM
Keywords: Domain Generalization, Vision-Language Models, Information Bottleneck, Conditional Entropy Bottleneck, Cross-modal Alignment

TL;DR

This work demonstrates that expressive visual guidance in vision-language distillation inevitably propagates domain-specific spurious cues, proposing a purely text-anchored Conditional Entropy Bottleneck framework that enforces domain invariance by distilling semantic anchors and compressing intra-class stylistic variations.

Background & Motivation

Visual recognition systems frequently suffer severe performance degradation when deployed into unseen target environments due to environmental distribution shifts. Unlike domain adaptation which benefits from unlabeled target domain data, domain generalization (DG) demands models to learn invariant representations preserving only task-relevant semantics without any target exposure. Recent methodologies heavily embrace large-scale vision-language models like CLIP, guided by the intuition that preserving their expressive visual representations directly promotes robustness under distribution shifts through knowledge distillation, prompt tuning, or weight ensembling.

However, does preserving expressive visual features genuinely benefit out-of-distribution generalization? Systematic empirical investigation reveals that highly expressive visual encoders inevitably entangle invariant core semantics with domain-specific visual styles (e.g., photo textures, sketch strokes, or artistic abstractions). Under finite model capacity constraints, visual guidance actively introduces spurious domain variations into the supervision signal, forcing models to expand their class representation bounds and blurring decision boundaries across environments. Conversely, text embedding spaces remain remarkably stable across diverse domains, exhibiting near-zero mutual information with domain identifiers while cleanly encoding class semantics.

Given that visual guidance inadvertently propagates domain entanglement, the most principled solution is to discard visual guidance altogether and position language embeddings as the primary anchor of domain invariance. Core idea: Anchor domain generalization supervision entirely to a frozen text embedding space, deriving a tractable Conditional Entropy Bottleneck (CEB) objective that maximizes task-relevant semantic mutual information while actively compressing domain-specific variations unshared with textual anchors.

Method

Overall Architecture

The proposed framework, Text-Anchored Information Bottleneck, introduces an architecture-agnostic approach that eliminates visual teachers. Training samples from multiple source domains are mapped by a trainable visual encoder \(f_\theta\) onto normalized hyperspherical features \(\mathbf{z} \in \mathbb{R}^d\). Concurrently, a frozen CLIP text encoder generates static semantic anchors \(T = [\mathbf{t}_1, \dots, \mathbf{t}_K]^\top \in \mathbb{R}^{K \times d}\) using standard label prompts such as "a photo of a [class]".

Governed by the Conditional Entropy Bottleneck (CEB) principle, the model optimizes two complementary mechanisms: instance-level semantic distillation \(\mathcal{L}_{\text{sem}}\) to maximize \(I(Z; Y)\), and batch-level bottleneck compression and alignment (\(\mathcal{L}_{\text{comp}}\) and \(\mathcal{L}_{\text{align}}\)) to minimize \(I(Z; X|Y)\). This dual formulation pulls image features toward their corresponding textual prototypes while forcing cross-domain intra-class samples to tightly cluster, effectively purifying the visual feature space from domain-specific styles.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Multi-source Domain Inputs X"] --> B["Trainable Visual Backbone f_theta"]
    B --> C["Hyperspherical Image Embeddings Z"]
    D["Frozen Text Encoder & Prompts"] --> E["Fixed Text Semantic Anchors T"]
    C --> F["Semantic Distillation<br/>Pull toward target anchor and push other classes"]
    E --> F
    C --> G["Intra-class Bottleneck Compression<br/>Maximize resultant vector length"]
    C --> H["Class-wise Directional Alignment<br/>Align mean feature with text anchor"]
    E --> H
    F --> I["Joint Objective Backpropagation<br/>Purify representations from domain styles"]
    G --> I
    H --> I

Key Designs

1. Semantic Distillation: Maximizing Class Semantic Sufficiency

Rather than relying on noisy cross-entropy loss against visual teachers, the framework maximizes the mutual information \(I(Z; Y)\) between learned representations \(Z\) and class labels \(Y\) by treating fixed text embeddings as deterministic class prototypes. For an input with label \(k\), the semantic distillation loss is defined as:

\[\mathcal{L}_{\text{sem}} = -\log \frac{\exp(\mathbf{t}_k^\top \mathbf{z} / \tau)}{\sum_{k'=1}^K \exp(\mathbf{t}_{k'}^\top \mathbf{z} / \tau)}\]

By pulling representations toward their assigned text anchor while repelling non-target anchors, this objective directly infuses the invariant semantic topology of the language space into the visual encoder, ensuring that the learned features retain core class identities uncorrupted by domain shifts.

2. Bottleneck Compression and Alignment: Suppressing Domain-Specific Variations

To minimize the conditional mutual information \(I(Z; X|Y)\) that carries spurious domain styles, the authors formulate a variational upper bound under the Markov chain assumption \(Y \leftrightarrow X \leftrightarrow Z\). Modeling prior \(q(Z|Y)\) and posterior \(p(Z|X)\) distributions on the unit hypersphere as von Mises-Fisher (vMF) distributions centered at text anchors \(\mathbf{t}_k\) and image features \(\mathbf{z}\), the minimization simplifies to maximizing the empirical expectation \(\mathbf{t}_k^\top \bar{\mathbf{z}}_k = \|\bar{\mathbf{z}}_k\| \cos(\mathbf{t}_k, \bar{\mathbf{z}}_k)\), where \(\bar{\mathbf{z}}_k = \frac{1}{|\mathcal{B}_k|}\sum_{i \in \mathcal{B}_k}\mathbf{z}_i\) is the batch-wise mean feature of class \(k\).

Because directly optimizing this combined inner product induces gradient coupling and early cosine saturation, the objective is decoupled into two independent surrogate terms: - Intra-class Compression Loss (\(\mathcal{L}_{\text{comp}}\)): Maximizes the resultant vector length \(\|\bar{\mathbf{z}}_k\|\), penalizing cross-domain angular dispersion and driving representations of the same class from different environments toward a compact, single-mode direction:

\[\mathcal{L}_{\text{comp}} = -\frac{1}{|\mathcal{K}_\mathcal{B}|}\sum_{k \in \mathcal{K}_\mathcal{B}} \|\bar{\mathbf{z}}_k\|\]

• Class-wise Directional Alignment Loss (\(\mathcal{L}_{\text{align}}\)): Maximizes the cosine similarity between the mean feature vector and its assigned text anchor, preventing degenerate representation collapse and locking semantic alignment:

\[\mathcal{L}_{\text{align}} = -\frac{1}{|\mathcal{K}_\mathcal{B}|}\sum_{k \in \mathcal{K}_\mathcal{B}} \frac{\mathbf{t}_k^\top \bar{\mathbf{z}}_k}{\|\bar{\mathbf{z}}_k\|}\]

Loss & Training

The total training objective integrates the three losses in a balanced multi-task configuration:

\[\mathcal{L} = \mathcal{L}_{\text{sem}} + \beta_1 \mathcal{L}_{\text{align}} + \beta_2 \mathcal{L}_{\text{comp}}\]

Across all benchmarks and backbones, the trade-off weights are fixed to \(\beta_1 = 0.1\) and \(\beta_2 = 1.0\), chosen via cross-validation on ResNet-50 trained on PACS. The text encoder remains frozen throughout optimization, significantly saving GPU memory and computational overhead.

Key Experimental Results

Main Results

Evaluated on the standard DomainBed benchmark suite using the leave-one-domain-out protocol averaged over three runs across ResNet-50, ViT-B/16, and CLIP-ViT-B/16 (original paper Table 1):

Backbone	Method	Guidance	VLCS	PACS	OfficeHome	TerraInc	DomainNet	Average
ResNet-50 (ImageNet-1k)	LP (Linear Probing)	-	78.1	86.2	68.4	46.3	41.2	64.0
ResNet-50	MIRO (ECCV'22)	-	79.0	85.4	70.5	50.4	44.3	65.9
ResNet-50	RISE (ICCV'23)	V, T	81.7	89.4	71.6	52.3	46.5	68.3
ResNet-50	VL2V (CVPR'24)	V, T	79.2	86.7	74.4	53.5	47.7	68.3
ResNet-50	Ours	T only	81.7	96.9	79.0	59.9	58.3	75.4 (+7.1)
ViT-B/16 (ImageNet-1k)	LP	-	79.5	81.5	82.8	42.2	50.5	67.3
ViT-B/16	RISE	V, T	84.2	91.0	80.3	44.6	56.6	71.3
ViT-B/16	VL2V	V, T	81.9	94.9	85.7	55.4	59.4	75.5
ViT-B/16	Ours	T only	86.2	94.1	86.4	62.2	68.8	79.5 (+4.0)
CLIP-ViT-B/16 (400M)	CLIP-ZeroShot	-	82.4	96.1	82.3	34.4	49.7	69.0
CLIP-ViT-B/16	CLIPood (ICML'23)	V, T	85.0	97.3	87.0	60.4	63.5	78.6
CLIP-ViT-B/16	CLIP-DTP (CVPR'25)	V, T	84.8	97.0	87.7	63.3	63.1	79.2
CLIP-ViT-B/16	Ours	T only	89.0	98.5	93.2	75.1	75.8	86.3 (+7.1)

On NICO++ evaluating background shift (original paper Table 2), the method elevates ResNet-50 performance from 82.5% to 95.7%, closing the gap to CLIP (97.5%), which underscores the model's capability to discard irrelevant contextual backgrounds.

Ablation Study

The contribution of each component within the text-anchored information bottleneck is ablated across OfficeHome, PACS, and DomainNet (original paper Table 3):

\(\mathcal{L}_{\text{sem}}\)	\(\mathcal{L}_{\text{align}}\)	\(\mathcal{L}_{\text{comp}}\)	OfficeHome (RN50 / ViT / CLIP)	PACS (RN50 / ViT / CLIP)	DomainNet (RN50 / ViT / CLIP)	Note
✓			73.4 / 82.4 / 85.2	92.6 / 92.3 / 96.8	35.5 / 52.8 / 60.3	Semantic distillation baseline
✓	✓		76.0 / 81.9 / 85.9	94.2 / 93.6 / 98.2	37.3 / 67.9 / 65.9	+ Directional alignment
✓		✓	78.9 / 85.1 / 90.8	95.8 / 94.0 / 98.1	57.6 / 68.7 / 75.2	+ Compression (largest single gain)
✓	✓	✓	79.0 / 86.4 / 93.2	96.9 / 94.1 / 98.5	58.3 / 68.8 / 75.8	Full model with complementary objectives

Key Findings

• Intra-class compression drives substantial OOD gains: Adding \(\mathcal{L}_{\text{comp}}\) to semantic distillation on DomainNet boosts ResNet-50 accuracy by +22.1% (35.5% \(\to\) 57.6%), confirming that explicitly suppressing intra-class stylistic variance is crucial for domain invariance.
• Increasing visual teacher guidance degrades robustness: Injecting CLIP image features into the distillation loss causes monotonic accuracy drops, with CNN architectures displaying higher sensitivity to visual noise than Transformers.
• Textual anchors provide general, model-agnostic invariance: Alternative text encoders (MiniLM achieving 79.1%, MPNet achieving 78.7% on OfficeHome) match CLIP text anchors (79.0%), and even fixed random anchors reach 76.6%, indicating that fixed geometrical anchors combined with compression form the foundational invariance mechanism.

Highlights & Insights

• Rethinking the role of visual teachers in DG: Contrary to the common assumption that richer visual teacher representations aid transfer learning, the paper illustrates that visual expressiveness carries domain styles that occupy limited model capacity and damage invariant generalization.
• Principled vMF Information Bottleneck Formulation: By casting the Conditional Entropy Bottleneck on the unit hypersphere with decoupled surrogate objectives, the paper provides a stable, elegant optimization routine that avoids gradient vanishing and representation collapse.
• Universal Architecture Robustness: Yields substantial gains across CNNs (ResNet-50, RegNet, EfficientNet) and Transformers (ViT, DeiT, Swin, DINOv2, CLIP) without requiring architecture-specific modifications.

Limitations & Future Work

• Risk in Context-Essential Tasks: By design, the information bottleneck purges background variations; in specialized domains where the background environment is intrinsic to object classification (e.g., specific wildlife habitats), aggressive filtering may discard helpful contextual cues.
• Static Text Template Constraints: The approach currently uses simple prompts ("a photo of a [class]"); extending this to adaptive, hierarchical, or descriptive language prompts could further enrich fine-grained semantic boundaries.
• Benchmark Pretraining Overlap: While tests on DINOv2 demonstrate genuine invariance learning, evaluating large foundational backbones under benchmarks that strictly guarantee zero pretraining domain overlap remains an ongoing challenge for the DG community.

Related Work & Insights

• vs RISE / VL2V: While existing VLM distillation frameworks treat CLIP's visual encoder as the primary teacher and text as auxiliary guidance, this work reveals visual guidance as a source of domain contamination and relies strictly on language embeddings.
• vs IIB / INSURE: Prior Information Bottleneck approaches in DG enforce variational compression towards uninformative Gaussian priors (\(\mathcal{N}(0, I)\)) or learn source-domain biased latent variables; this work introduces label-conditional spherical vMF priors anchored to immutable language semantics.
• vs CLIPood / Robust Fine-Tuning: Prior methods focus primarily on preserving original CLIP weights, which limits applicability to non-CLIP backbones; the proposed text-anchored CEB framework is universally applicable to any visual architecture.

Rating

• Novelty: ⭐⭐⭐⭐⭐ Offers a refreshing, theoretically grounded paradigm shift away from visual teacher reliance in domain generalization.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ Extensively validated across 6 benchmarks, 7 diverse backbones, and supported by mutual information analysis and Lipschitz stability diagnostics.
• Writing Quality: ⭐⭐⭐⭐⭐ Clear exposition, logically rigorous narrative flow, and well-structured mathematical derivations.
• Value: ⭐⭐⭐⭐⭐ Provides a simple, highly effective recipe for out-of-distribution robustness in multimodal and visual representation learning.

Related Papers

• [CVPR 2026] Towards Multimodal Domain Generalization with Few Labels
• [ICLR 2026] Calibrated Information Bottleneck for Trusted Multi-modal Clustering
• [ACL 2026] From Verbatim to Gist: Distilling Pyramidal Multimodal Memory via Semantic Information Bottleneck
• [ICML 2025] Learning Optimal Multimodal Information Bottleneck Representations
• [AAAI 2026] Conditional Information Bottleneck for Multimodal Fusion: Overcoming Shortcut Learning in Sarcasm Detection