Skip to content

DINOde: Continuous Vision-Text Alignment for Open-Vocabulary Semantic Segmentation

Conference: ECCV 2026
Paper: ECCV Official
Code: https://github.com/yoon307/DINOde
Area: Segmentation
Keywords: Open-Vocabulary Semantic Segmentation, Neural ODE, DINOv3, Cross-Modal Manifold Alignment, Tangent Space Projection

TL;DR

DINOde introduces an ordinary differential equation (ODE)-based continuous cross-modal alignment framework that smoothly maps frozen CLIP text embeddings onto the self-supervised DINOv3 visual manifold via Semantic Text Flow, Global Context Flow, and Velocity Tangent Projection, achieving state-of-the-art results across 8 open-vocabulary benchmarks after just 4 hours of training on COCO Captions.

Background & Motivation

Open-vocabulary semantic segmentation (OVSS) aims to identify and delineate visual concepts beyond predefined fixed categories by leveraging open natural language descriptions. Existing methodologies predominantly build upon large-scale vision-language models like CLIP. However, because CLIP's vision backbone is optimized primarily for global image-text contrastive alignment, its spatial patch representations are coarse, blurry, and spatially entangled, rendering them poorly suited for dense pixel-level prediction. In contrast, self-supervised vision models (SSVMs) such as DINOv2 and DINOv3 excel at capturing emergent object-centric layouts and fine spatial boundaries, yet their lack of inherent textual alignment prevents direct zero-shot open-vocabulary transfer.

To bridge this modality gap, earlier attempts have either resorted to training an entire text encoder from scratch on vast datasets (e.g., dino.txt) or introduced multi-layer perceptron (MLP) projectors to map text representations into the visual feature space. Nevertheless, such instantaneous, single-step Euclidean mappings completely overlook the intrinsic Riemannian curvature of the hyperspherical representation manifold. Forcing heterogeneous features together via unconstrained discrete projections induces severe manifold entanglement, disrupting geodesic distances and neighbor topology among semantic concepts and degrading downstream segmentation accuracy.

This work departs from rigid static projections by formulating cross-modal alignment as a continuous dynamical system. The core idea is to model vision-text alignment as a continuous ordinary differential equation (ODE) trajectory under hyperspherical geometric constraints, utilizing Semantic Text Flow and Global Context Flow to smoothly guide text and holistic visual representations toward the DINO manifold while strictly preserving feature topology.

Method

Overall Architecture

DINOde retains both the DINOv3 visual backbone (ViT-L/16) and the CLIP text encoder (ViT-L/14) completely frozen, training only lightweight continuous velocity fields parameterized by neural networks. Given an input image, DINOv3 produces spatial patch tokens alongside a holistic [CLS] token, while candidate text prompts are encoded into semantic text embeddings. The architecture coordinates two dynamical flows under a geometric constraint: Semantic Text Flow (STF) transports text embeddings toward the visual manifold; Global Context Flow (GCF) continuously refines the [CLS] token to preserve global scene semantics; and Velocity Tangent Projection (VTP) ensures that the velocity vectors adhere strictly to the tangent space of the unit hypersphere.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input Image & Text Prompts"] --> B["Frozen Feature Extraction: DINOv3 Patch/CLS + CLIP Text"]
    B --> C["Semantic Text Flow (STF)<br/>Continuous ODE transforms text along hypersphere"]
    B --> D["Global Context Flow (GCF)<br/>Continuous ODE refines holistic [CLS] token"]
    C --> E["Velocity Tangent Projection (VTP)<br/>Projects velocity to tangent plane, removing radial drift"]
    D --> E
    E --> F["Dual-Branch Feature Concatenation<br/>Top-K pooled patches + GCF-refined CLS representation"]
    F --> G["Symmetric InfoNCE Contrastive Objective (Trained on 118k COCO Captions)"]
    G --> H["Test-time Inference: Precomputed ODE text anchors matched with DINO patches"]

Key Designs

1. Semantic Text Flow: Progressive Manifold Transfer via Continuous ODEs To overcome the topological deformation caused by discrete one-step MLP mappings, Semantic Text Flow (STF) formulates alignment as an initial value problem governed by \(\frac{dz_t}{dt} = v_\theta(z_t, t)\) over \(t \in [0, 1]\). The initial state \(z_0 = \mathrm{Norm}(W z^{\text{text}})\) is obtained by mapping CLIP text embeddings to DINO's feature dimension followed by \(\ell_2\) normalization. The velocity network \(F_\theta\) is conditioned on virtual integration time \(t\) via sinusoidal modulation \(\gamma(t)\). Using an explicit Euler solver with step size \(\Delta t = 1/N_{\text{step}}\) (default \(N_{\text{step}} = 10\)), the state evolves smoothly: $\(z_{t_{k+1}} = \mathrm{Norm}\big(z_{t_k} + \Delta t \cdot \tilde{v}_\theta(z_{t_k}, t_k)\big)\)$ The final state \(\hat{z}^{\text{text}} = z_1\) provides an aligned semantic anchor located stably on the visual manifold, preserving neighborhood relationships without Euclidean distortion.

2. Velocity Tangent Projection: Geometry-Preserving Hyperspherical Constraint Because contrastive representations reside on the unit hypersphere \(\mathbb{S}^{D-1}\), unconstrained velocity fields pointing inward or outward force intermediate states away from the manifold, where naively reapplying \(\ell_2\) normalization induces irregular stretching and nonlinear distortion. Velocity Tangent Projection (VTP) eliminates this artifact by projecting the raw velocity field onto the tangent space of the current state \(z_t\) before integration: $\(\tilde{v}_\theta(z_t, t) = v_\theta(z_t, t) - \langle v_\theta(z_t, t), z_t \rangle z_t\)$ By removing the radial component orthogonal to the hypersphere surface, VTP guarantees that feature updates travel along geodesic paths, enforcing norm conservation and preserving the underlying Riemannian geometry.

3. Global Context Flow: Multi-Granularity Holistic-Local Coordination Relying solely on local patch alignment risks overfitting the text trajectory to salient visual details while overlooking holistic scene-level semantics. To counterbalance this, Global Context Flow (GCF) refines DINO's raw [CLS] token \(z^{\text{img}}_{\text{cls}}\). Starting from \(c_0 = \mathrm{Norm}(W_{\text{cls}} z^{\text{img}}_{\text{cls}})\), a dedicated velocity network \(u_\phi(c_t, t)\) integrates the representation forward: $\(c_{t_{k+1}} = \mathrm{Norm}\big(c_{t_k} + \Delta t \cdot \tilde{u}_\phi(c_{t_k}, t_k)\big)\)$ During training, the overall image embedding is formed by concatenating the Top-K pooled patch descriptor \(z^{\text{img}}\) with the GCF-refined global token \(\hat{z}^{\text{img}}_{\text{cls}} = c_1\): $\(z^{\text{img}}_{||} = \big[ z^{\text{img}} \;;\; \hat{z}^{\text{img}}_{\text{cls}} \big] \in \mathbb{R}^{2D}\)$ Text embeddings are duplicated to match dimensionality. This joint objective aligns text anchors to both fine-grained patch clusters and overarching scene context, significantly boosting semantic disambiguation.

Loss & Training

The network is trained with a symmetric InfoNCE contrastive loss over minibatches of size \(B\): $\(\mathcal{L}_{\text{NCE}} = \frac{1}{2} \Big( \mathrm{CE}(S, y) + \mathrm{CE}(S^\top, y) \Big)\)$ where \(S_{ij} = \frac{1}{\tau} (z^{\text{img}}_{||, i})^\top z^{\text{text}}_{||, j}\) denotes pairwise cosine similarities scaled by temperature \(\tau = 0.07\). Training requires only the COCO 2017 Caption dataset (~118k images), optimized via AdamW (learning rate \(1 \times 10^{-4}\), weight decay 0.01) for 20 epochs in under 4 hours on a single RTX 3090 GPU. At test time, category embeddings are evolved through STF once and cached offline, requiring only dot-product similarity against image patch tokens during inference.

Key Experimental Results

Main Results

Evaluation spans 8 competitive OVSS benchmarks across datasets with background categories (Pascal VOC 21, Pascal Context 60, COCO Object) and without background categories (Pascal VOC 20, Pascal Context 59, COCO Stuff, Cityscapes, ADE20K). The table below summarizes main results without post-processing mask refinement (Table 5 in the original paper):

Method Visual Encoder V20 C59 Stuff City ADE V21 C60 Object Avg mIoU
MaskCLIP (ECCV22) CLIP ViT-L 29.4 12.4 8.8 11.5 7.2 23.3 11.7 7.2 13.9
FreeDA (CVPR24) CLIP+DINOv2 85.7 39.7 26.3 33.6 21.4 44.1 34.8 33.9 39.9
ProxyCLIP (ECCV24) CLIP+DINO 83.2 37.7 25.6 40.1 22.6 60.6 34.5 39.2 43.0
Talk2DINO (ICCV25) DINOv2(reg) 87.1 39.1 27.0 35.8 21.1 60.1 34.2 37.6 42.8
Talk2DINO* (ICCV25) DINOv3 ViT-L 87.7 43.0 32.6 40.8 24.2 65.9 37.9 48.6 47.6
DINOde (Ours) DINOv3 ViT-L 91.1 44.5 31.6 45.4 25.4 69.2 40.7 48.0 49.5

When augmented with Pixel-Adaptive Mask Refinement (PAMR), DINOde reaches an average mIoU of 50.1%, setting new state-of-the-art marks on 6 out of 8 benchmark datasets.

Ablation Study

The component ablation (Table 1 in the paper) highlights the individual and collective impact of STF, VTP, and GCF compared against an iso-parameter MLP mapping baseline:

Config STF VTP GCF V20 C59 Stuff City ADE V21 C60 Object Avg mIoU
(a) MLP Baseline - - - 88.4 43.2 29.7 42.5 23.2 68.0 39.6 46.7 47.7
(b) STF only ✓ - - 88.4 43.0 31.2 44.7 25.2 66.6 39.5 47.1 48.2
(c) STF + VTP ✓ ✓ - 88.5 43.4 31.4 45.4 25.5 67.3 39.6 47.2 48.5
(d) STF + GCF ✓ - ✓ 90.2 44.3 31.3 45.6 24.8 69.5 40.3 47.2 49.2
(e) Full DINOde ✓ ✓ ✓ 91.1 44.5 31.6 45.4 25.4 69.2 40.7 48.0 49.5

Ablating the integration step count \(N_{\text{step}}\) (Table 2 in the paper) reveals that \(N=5\) already yields 48.7%, \(N=10\) achieves the optimal 49.5%, and \(N=50\) reaches 49.6% with marginal plateauing, confirming \(N=10\) as the best speed-accuracy trade-off.

Key Findings

  • Continuous trajectories systematically beat discrete projections: Moving from static MLP projection (47.7%) to the full ODE flow (49.5%) yields a +1.8%p improvement across benchmarks, demonstrating that gradual manifold evolution mitigates catastrophic feature distortion.
  • Superior manifold geometry preservation: Table 4 verifies that STF consistently outperforms MLP across four geometric criteria: top-10 nearest neighbor overlap (0.601 vs. 0.575), geodesic consistency correlation (0.712 vs. 0.693), class-structure CKA (0.871 vs. 0.843), and alignment compactness (0.423 vs. 0.412).
  • Strong backbone agnostic generalization: DINOde consistently outperforms MLP baselines when scaling down DINOv3 to ViT-B (46.1% vs. 44.2%) and ViT-S (38.8% vs. 36.9%), or switching CLIP to ViT-B (48.7% vs. 47.8%).

Highlights & Insights

  • Geometric rigor via tangent projection: Constraining vector fields to the hypersphere tangent space provides an elegant and mathematically sound method to prevent radial drift and preserve Riemannian distance metrics in contrastive spaces.
  • Extreme sample and compute efficiency: By avoiding text encoder retraining from scratch, DINOde achieves state-of-the-art OVSS performance with only ~118k caption pairs and 4 hours on a single commodity GPU, drastically lowering the resource requirements for cross-modal adaptation.
  • Broader implications for MLLM projector design: Modern multimodal LLMs rely almost universally on simple MLP or linear projectors to interface visual tokens with language models. DINOde's findings suggest that replacing discrete projectors with continuous ODE-based flows could substantially improve visual grounding and prevent semantic collapsing in MLLMs.

Limitations & Future Work

  • Inference latency for novel online text: While fixed class anchors can be cached offline, dynamic, streaming textual queries incur the runtime cost of 10 numerical Euler integration steps.
  • ODE solver accumulation error: Higher step counts (\(N=50\)) occasionally display minor degradation on specific benchmarks due to numerical accumulation error; exploring higher-order or adaptive solvers (e.g., RK4) remains an open challenge.
  • Extension to instance-level tasks: Extending the continuous flow formulation to open-vocabulary instance and panoptic segmentation, as well as multimodal LLM token projections, offers promising future research avenues.
  • vs Talk2DINO: Talk2DINO aligns DINOv2 self-attention heads with CLIP via static nonlinear transformations. DINOde demonstrates that static mappings deform manifold curvature and demonstrates a 1.9%p average mIoU advantage under the identical DINOv3 backbone.
  • vs dino.txt: dino.txt trains a custom text encoder from scratch on large-scale paired data, demanding heavy compute. DINOde leverages frozen CLIP weights and lightweight velocity fields to achieve superior performance with vastly reduced computational requirements.

Rating

  • Novelty: ⭐⭐⭐⭐☆ Translates continuous neural ODE dynamics and Riemannian tangent projections into cross-modal vision-language alignment for dense prediction.
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ Rigorous evaluation across 8 benchmarks, granular component ablations, step-count sweeps, cross-backbone testing, and quantitative geometric diagnostics.
  • Writing Quality: ⭐⭐⭐⭐⭐ Well-structured, mathematically concise, and clearly illustrated with meaningful empirical analyses.
  • Value: ⭐⭐⭐⭐⭐ Provides an efficient, geometry-preserving template for bridging self-supervised visual encoders with language models without heavy retraining.