Hierarchical Prompt Injector for Domain Generalization Segmentation¶

Conference: ECCV 2026
Paper: ECCV 2026 Official
Code: https://github.com/MosukFate/HPI
Area: Segmentation
Keywords: Domain Generalized Semantic Segmentation, Vision-Language Models, Spatial Hierarchical Prompts, Adaptive Prompt Injection, DINOv2

TL;DR¶

Addressing diffuse attention and negative transfer caused by holistic category-level prompts and uniform injection in domain generalized semantic segmentation, this paper proposes Spatial Hierarchical Prompts (SHP) and the Hierarchical Prompt Injector (HPI) with dual-confidence gating and alignment losses, achieving state-of-the-art synthetic-to-real performance of 70.62% mIoU.

Background & Motivation¶

Domain Generalized Semantic Segmentation (DGSS) aims to train segmentation models solely on labeled source domains such that they generalize effectively to unseen target domains without accessing target distribution data during training. Conventional DGSS paradigms primarily rely on low-level statistical manipulations—such as style augmentation, texture diversification, or frequency-domain perturbations—to reduce reliance on domain-specific appearance cues. However, these techniques operate purely on superficial surface statistics, neglecting the invariant geometric and structural configurations inherent to physical objects. In contrast to texture and illumination, topological and geometric attributes exhibit superior stability across environmental domain shifts, prompting recent research to transfer structural priors through Vision-Language Models (VLMs, such as CLIP).

Nonetheless, current prompt-learning formulations for DGSS suffer from two fundamental bottlenecks. First, existing methods typically compress each semantic category into a single holistic sentence (such as class names or brief domain descriptors). This global compression erases region-level geometric diversity and generates diffuse attention maps that bleed into irrelevant background areas. In practical real-world driving scenes, viewpoint variations and severe occlusions routinely obscure significant portions of an object, rendering a single monolithic description incapable of matching partially visible visual cues. Second, existing frameworks either inject prompts solely into the decoder stage—leaving visual encoder features entirely unguided—or broadcast textual prompts uniformly across all encoder patch tokens. Lacking spatial selectivity and explicit prompt-to-region correspondence supervision, such uniform modulation inevitably introduces spurious feature updates in background regions, which is further exacerbated by the coarse patch-alignment capability of contrastively pretrained VLMs.

To overcome these challenges, prompt engineering must move from monolithic class descriptions to structured regional anchors, coupled with spatially gated injection. The core idea is to decompose each category into multi-view Spatial Hierarchical Prompts (SHP) that serve as stable regional geometric anchors, and introduce a Hierarchical Prompt Injector (HPI) governed by Semantic Alignment Confidence (SAC) and Spatial Coherence Confidence (SCC) along with explicit spatial and semantic auxiliary supervision, achieving fine-grained, content-aware modulation of frozen foundation features.

Method¶

Overall Architecture¶

HPI utilizes a dual-foundation-model paradigm to combine rich cross-modal semantics with fine-grained spatial localization. An input image is concurrently processed by a frozen VLM image encoder (e.g., CLIP-ViT-L or EVA02-L) as the primary backbone and a frozen DINOv2 encoder. Features from DINOv2 are continuously integrated into the VLM stream via lightweight linear adapters to reinforce pixel-level structural fidelity. At deep layers (specifically layer 23), offline LLM-generated and VLM-encoded Spatial Hierarchical Prompts (SHP) act as queries in a cross-attention mechanism with visual patch tokens. Rather than directly injecting text embeddings, the cross-attention matrix is reorganized into a patch-to-patch routing operator that modulates visual values among prompt-relevant patches. Subsequently, two complementary confidence maps—Semantic Alignment Confidence (SAC) and Spatial Coherence Confidence (SCC)—adaptively gate the generated residuals to prevent background contamination and spatial fragmentation. Multi-scale feature representations from both encoders are concatenated and fed into a Mask2Former decoder to predict final segmentation masks, optimized jointly by standard segmentation losses and two auxiliary prompt-alignment objectives.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input Image I"] --> B["Frozen Foundation Encoders<br/>VLM Backbone + DINOv2 Spatial Stream"]
    C["Spatial Hierarchical Prompts<br/>LLM-Crafted 4-Component Prompts (R=3)"] --> D["Hierarchical Prompt Injector<br/>Cross-Attention Patch Routing (Layer 23)"]
    B --> D
    D --> E["Adaptive Dual-Confidence Gating<br/>Semantic SAC + Spatial SCC"]
    E --> F["Auxiliary Alignment Losses & Decoding<br/>L_spatial + L_semantic + Mask2Former"]
    F --> G["Domain Generalized Segmentation"]

Key Designs¶

1. Spatial Hierarchical Prompts: Decomposing Multi-View Regional Geometric Anchors

To address the failure of holistic class prompts under viewpoint variation and partial occlusion, this design decomposes each category into \(R\) (default \(R=3\)) complementary region-level prompts \(\mathcal{D}_i = \{d_i^{(r)}\}_{r=1}^R\), representing canonical spatial views (e.g., front, side, and rear of a vehicle). Each regional prompt is structured with four complementary components: [Region Cue] + [Part] + [Geometry] + [Boundary Relation]. For instance, a rider prompt is formulated as: "Head forms round helmet above handlebar boundary". Here, the region cue anchors the viewing perspective for inference-time selection; the part and geometry components specify domain-invariant rigid structures; and the boundary relation captures topological interactions with adjacent semantic entities. All \(K = C_{\text{cls}} \times R\) prompts are synthesized offline using an LLM and encoded with the frozen VLM text encoder \(E_T\) to produce structural prompt features \(f_{\text{SHP}} \in \mathbb{R}^{K \times D}\). These features are zero-padded to match the channel dimensions of the VLM image backbone, while for DINOv2, they are mapped via a pretrained Talk2DINO projection layer to bridge the vision-only representation gap.

2. Hierarchical Prompt Injector: Patch Routing via Dual-Confidence Adaptive Gating

To prevent feature magnitude distortion caused by directly injecting textual values and avoid negative transfer from uniform broadcasting, HPI establishes a prompt-induced patch routing mechanism. For each branch \(x \in \{\text{vlm}, \text{dino}\}\), intermediate visual patch tokens \(\mathbf{P}^x \in \mathbb{R}^{B \times N \times C}\) and prompt embeddings \(f_{\text{SHP}}^x\) undergo cross-attention to produce attention affinity maps \(\mathbf{A}^x = \text{CrossAttn}(f_{\text{SHP}}^x, \mathbf{P}^x) \in \mathbb{R}^{B \times K \times N}\). The prompt-conditioned residual update is formed via visual value aggregation:

\[\Delta \mathbf{P}^x = (\mathbf{A}^x)^\top \mathbf{A}^x \mathbf{V}^x \in \mathbb{R}^{B \times N \times C}\]

Here, \((\mathbf{A}^x)^\top \mathbf{A}^x\) functions as a patch routing matrix connecting patches that co-activate on the same regional prompt, routing visual information rather than raw text vectors. To prevent residual leakage caused by the non-zero nature of softmax over background patches, HPI applies two complementary confidence weight maps: - Semantic Alignment Confidence (SAC): Derived by taking the maximum head-averaged attention weight across all \(K\) prompts for each patch, \(\mathbf{w}_{\text{SAC}, b, n} = \max_{k \in [K]} \bar{\mathbf{A}}^x_{b, n, k} \in [0, 1]\), suppressing residual injection at positions lacking semantic relevance. - Spatial Coherence Confidence (SCC): To prevent isolated high-SAC noisy activations (e.g., wheels shared between cars and trucks), the concatenated features \([\mathbf{P}^x, \Delta \mathbf{P}^x]\) are projected, reshaped to a 2D spatial grid, and processed by parallel depthwise/pointwise convolutions with kernel sizes \(1\times 1\), \(3\times 3\), and \(5\times 5\), followed by fusion and a sigmoid function to output contiguous spatial weights \(\mathbf{w}_{\text{SCC}} \in [0, 1]^{B \times N \times 1}\).

A learnable scalar \(\beta = \sigma(\beta_0)\) (initialized at 0.5) balances both confidences to modulate the final visual representations:

\[\mathbf{P}'^{\ , x} = \mathbf{P}^x + [\beta \mathbf{w}_{\text{SAC}} + (1 - \beta) \mathbf{w}_{\text{SCC}}] \odot \Delta \mathbf{P}^x\]

3. Alignment Auxiliary Supervision: Explicit Spatial Localization and Semantic Consistency Regularization

To rectify the lack of native patch-level alignment in contrastively pretrained VLMs, this design introduces two auxiliary supervision losses directly on the attention weights \(\bar{\mathbf{A}}^x\). First, the Spatial Localization Loss (\(\mathcal{L}_{\text{spatial}}\)) constrains the attention maps using ground-truth segmentation masks \(y_{\text{seg}}\). For each class \(c\), the attention scores of its \(R\) regional prompts are aggregated across patches via a Log-Sum-Exp (LSE) operator and normalized across all classes via softmax to form a patch-level distribution \(\boldsymbol{\pi}^x \in \mathbb{R}^{B \times C_{\text{cls}} \times H \times W}\):

\[\boldsymbol{\pi}_c^x(n) = \frac{\exp\left(\mathrm{LSE}_{r \in \mathcal{K}(c)}(\log \bar{\mathbf{A}}_{n, r}^x)\right)}{\sum_{c'} \exp\left(\mathrm{LSE}_{r \in \mathcal{K}(c')}(\log \bar{\mathbf{A}}_{n, r}^x)\right)}\]

The distribution is upsampled to the original spatial resolution and supervised by \(y_{\text{seg}}\) via negative log-likelihood (NLL). Second, the Semantic Consistency Loss (\(\mathcal{L}_{\text{semantic}}\)) addresses the limitation that softmax normalization cannot drive absent class activations to absolute zero. Using the binary image-level presence vector \(\mathbf{y}_{\text{cls}} \in \{0, 1\}^{B \times C_{\text{cls}}}\), attention-weighted visual features are pooled per prompt as \(\mathbf{v}_{b, k}^x = \sum_{n=1}^N \mathbf{A}_{b, n, k}^x \mathbf{P}_{b, n}^x\). The cosine similarity between \(\mathbf{v}_{b, k}^x\) and prompt embedding \(f_{\text{SHP}, k}^x\) is calculated and mean-pooled across the \(R\) prompts to predict class presence logits \(\hat{\mathbf{s}}^x \in \mathbb{R}^{B \times C_{\text{cls}}}\), optimized with binary cross-entropy (BCE) to strictly penalize activations of absent classes.

Loss & Training¶

The overall training objective combines the standard segmentation loss with the two auxiliary alignment objectives:

\[\mathcal{L}_{\text{total}} = \mathcal{L}_{\text{seg}} + \lambda_{\text{spatial}} \mathcal{L}_{\text{spatial}} + \lambda_{\text{semantic}} \mathcal{L}_{\text{semantic}}\]

where \(\mathcal{L}_{\text{seg}}\) represents the standard Mask2Former cross-entropy and Dice loss combination. The loss weights are set to \(\lambda_{\text{spatial}} = 0.1\) and \(\lambda_{\text{semantic}} = 0.05\). All backbone foundation models are frozen; only the HPI modules, linear fusion adapters, and Mask2Former decoder are optimized. The framework is trained for 12k iterations with an AdamW optimizer, a learning rate of 1e-4, batch size 2, and a linear warm-up over 1.5k iterations on a single NVIDIA RTX 3090 GPU.

Key Experimental Results¶

Main Results¶

Under the standard synthetic-to-real (GTA5 → Cityscapes, BDD100K, Mapillary, denoted as G→{C, B, M}) and real-to-real (Cityscapes → BDD100K, Mapillary, denoted as C→{B, M}) protocols, 19-class mIoU results are summarized below:

Evaluation Protocol	Backbone	Ours (HPI)	Prev. SOTA / Strong Baselines (MFuser / SoMA)	Gain
G → Cityscapes (C)	CLIP-L + DINOv2-L	74.05	71.24 (MFuser) / 71.82 (SoMA)	+2.81 / +2.23
G → BDD100K (B)	CLIP-L + DINOv2-L	63.14	61.08 (MFuser) / 61.31 (SoMA)	+2.06 / +1.83
G → Mapillary (M)	CLIP-L + DINOv2-L	74.67	71.14 (MFuser) / 71.67 (SoMA)	+3.53 / +3.00
G → {C, B, M} Avg.	CLIP-L + DINOv2-L	70.62	67.82 (MFuser) / 68.27 (SoMA)	+2.80 / +2.35
G → {C, B, M} Avg.	EVA02-L + DINOv2-L	69.39	68.20 (MFuser) / 66.09 (FADA)	+1.19 / +3.30
C → BDD100K (B)	EVA02-L + DINOv2-L	66.23	65.81 (MFuser) / 67.02 (SoMA)	+0.42 / -0.79
C → Mapillary (M)	EVA02-L + DINOv2-L	79.25	77.93 (MFuser) / 76.45 (SoMA)	+1.32 / +2.80
C → {B, M} Avg.	EVA02-L + DINOv2-L	72.74	71.87 (MFuser) / 71.74 (SoMA)	+0.87 / +1.00
C → {B, M} Avg.	CLIP-L + DINOv2-L	72.73	71.84 (MFuser) / 68.65 (Rein)	+0.89 / +4.08

On the challenging SYNTHIA → {C, B, M} benchmark, HPI (CLIP-L) achieves an average mIoU of 52.72%, outperforming MFuser (50.34%) by +2.38%, with a notable +3.60% improvement on BDD100K (49.19% vs. 45.59%).

Ablation Study¶

Ablation experiments on G → {C, B, M} using the CLIP-L + DINOv2-L architecture highlight the individual contributions of each component:

Config / Component	G→C mIoU	G→B mIoU	G→M mIoU	Avg. mIoU	Note
Ours (R=3, full model)	74.05	63.14	74.67	70.62	Full HPI framework
Prompt count R=2	72.68	62.26	73.49	69.48	Incomplete viewpoint coverage (-1.14%)
Prompt count R=4	73.01	62.58	73.48	69.69	Overlapping prompts dilute selectivity (-0.93%)
w/o Region Cue	72.33	62.24	73.26	69.28	Viewpoint anchor missing, largest drop (-1.34%)
w/o Part	72.49	62.40	73.30	69.40	Discriminative sub-structure absent (-1.22%)
w/o Geometry	72.76	62.40	73.17	69.44	Loss of domain-invariant shape priors (-1.18%)
w/o Boundary Relation	72.96	62.50	73.77	69.74	Contextual boundary grounding missing (-0.88%)
w/o SAC	73.83	61.61	73.76	69.73	Unchecked semantic leakage (-0.89%)
w/o SCC	73.17	62.63	74.03	69.94	Spatial noise and patch fragmentation (-0.68%)
w/o \(\mathcal{L}_{\text{spatial}} + \mathcal{L}_{\text{semantic}}\)	72.51	62.69	73.01	69.40	Absence of alignment supervision (-1.22%)
w/o \(\mathcal{L}_{\text{spatial}}\)	72.99	62.07	73.56	69.54	Spatial grounding drift (-1.08%)
w/o \(\mathcal{L}_{\text{semantic}}\)	73.10	62.87	73.76	69.91	Inability to suppress absent classes (-0.71%)

Key Findings¶

Crucial Role of Viewpoint Anchors: Ablating the Region Cue component induces the largest drop among all prompt elements (-1.34% mIoU), proving that establishing viewpoint orientation is essential for cross-attention to resolve local geometric patterns under severe domain shifts. Setting \(R=3\) achieves optimal trade-off between coverage and prompt competition.
Synergy of Dual-Confidence Gating: SAC filters out semantically irrelevant background patches (removing SAC drops G→B mIoU by 1.53%), while SCC enforces contiguous multi-scale spatial neighborhood agreement. Together, they eliminate isolated patch noise and negative transfer.
Deep-Layer Injection Superiority: Sweeping injection layers demonstrates monotonic performance improvements from Layer 5 (69.32%) to Layer 23 (70.62%). Shallow layers capture low-level, domain-fragile textures that fail to align with semantic prompts, whereas deep layers offer mature feature representations that benefit maximally from structural geometric priors.

Highlights & Insights¶

From Monolithic Classes to View-Decomposed Anchors: Instead of compressing complex categories into single global text embeddings, decomposing classes into four-component regional prompts (view, part, geometry, boundary) ensures robust geometric matching even when objects are partially occluded.
Patch-to-Patch Routing via Cross-Attention: The \((\mathbf{A}^x)^\top \mathbf{A}^x \mathbf{V}^x\) reformulation uses textual prompts to steer visual feature aggregation directly among related patches, avoiding cross-modal dimension mismatches and feature distortion.
Dual-Confidence Adaptive Filtering: Combining attention-derived semantic filtering (SAC) with multi-scale convolutional spatial smoothing (SCC) provides a generalizable mechanism to gate prompt injection in dense prediction tasks.

Limitations & Future Work¶

Reliance on LLM Priors: The generation of multi-component prompts relies on structured prompting with frozen LLMs. For niche domains or fine-grained industrial tasks lacking extensive pretraining data, prompt descriptions may require manual calibration.
Computational Overhead of Dual Encoders: Although foundation models remain frozen, running both VLM and DINOv2 encoders in parallel incurs additional memory and inference latency on edge deployment platforms.
Future Directions: Exploring end-to-end differentiable, dynamic prompt generation conditioned on visual scene context, and distilling the multi-model teacher representation into a lightweight single-backbone student model.

vs MFuser [CVPR 2025]: While MFuser uses Mamba blocks to bridge VLM and DINOv2 globally, it applies uniform fusion. HPI introduces spatial hierarchical prompts with patch-adaptive gating, improving synthetic-to-real performance by +2.80% mIoU.
vs TQDM [ECCV 2024] / DenseCLIP [CVPR 2022]: Earlier methods restricted text injection to decoder queries or uniform encoder additions. HPI modulates deep encoder features adaptively, supervised by explicit spatial and semantic auxiliary losses to eliminate background hallucinations.
vs SoMA [CVPR 2025] / Rein [CVPR 2024]: These PEFT approaches tune vision-only foundation models via parameter adapters without linguistic guidance. HPI demonstrates that structured cross-modal geometric prompts can significantly outperform purely visual parameter-efficient tuning (+2.35% mIoU over SoMA).

Rating¶

Novelty: ⭐⭐⭐⭐⭐ [Decoupling categories into multi-view four-component geometric prompts combined with adaptive patch routing is conceptually elegant and effective.]
Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Extensive evaluations across 5 datasets, 3 DGSS settings, multiple foundation models, and exhaustive component ablations.]
Writing Quality: ⭐⭐⭐⭐⭐ [Clear motivation, rigorous mathematical formulation, and well-structured empirical analysis.]
Value: ⭐⭐⭐⭐⭐ [Sets a new benchmark for foundation-model adaptation in domain generalized dense visual perception with open-source code.]