Learning Structured Visual Compositional Representations for Weakly Supervised Referring Expression Comprehension¶
Conference: ECCV 2026
Paper: ECCV 2026 Poster
Code: https://github.com/quiet5daxis/SVCR
Area: Others
Keywords: Weakly Supervised REC, Structured Visual Representation, Compositional Alignment, Unary Object Representation, Pairwise Relational Representation
TL;DR¶
Addressing the representation mismatch where weakly supervised referring expression comprehension (WREC) flattens relational context into unary anchors, SVCR explicitly constructs a structured visual representation space with decoupled unary object representations and pairwise relational representations, achieving state-of-the-art grounding accuracy via multi-granular compositional alignment without bounding-box annotations.
Background & Motivation¶
Referring Expression Comprehension (REC), also termed visual grounding, aims to localize target objects within an image based on natural language descriptions. It serves as a foundational capability for embodied AI applications such as human-robot interaction and visual navigation. While fully supervised methods achieve high accuracy via dense bounding-box annotations, the immense annotation overhead and poor cross-domain transferability have shifted research focus toward weakly supervised REC (WREC), which trains models using only paired image-text descriptions. Modern one-stage WREC pipelines predominantly operate under contrastive anchor-text matching paradigms, iteratively identifying pseudo-positive anchors to guide Expectation-Maximization (EM)-like parameter optimization, frequently enriched with visual foundation models or geometric priors.
However, existing WREC methodologies suffer from an intrinsic structural defect: both visual representations and cross-modal alignments are confined strictly to the unary anchor level. Even when inter-object relational cues or graph propagation are incorporated, contextual features are ultimately aggregated and absorbed back into individual anchor vectors, leaving the visual representation space structurally flat. In sharp contrast, natural referring expressions are inherently compositional and relational: they identify referents not merely by intrinsic unary attributes (e.g., "black", "wooden") but prominently through pairwise spatial and semantic relations (e.g., "on the table", "to the left of the man"). Compressing multi-object relational interactions into isolated unary embeddings entangles relation semantics with single-object appearance and category features, undermining the model's capacity to resolve complex compositional expressions.
This work breaks through the flat unary constraint by explicitly mirroring the compositional structure of referring language directly within the visual space. Core idea: Decouple and construct a structured visual representation space comprising Unary Object Representations (UOR) and Pairwise Relational Representations (PRR), enabling multi-granular compositional alignment with subject noun chunks and full-sentence embeddings alongside hierarchical consistency constraints under weak supervision.
Method¶
Overall Architecture¶
The SVCR framework comprises multi-source feature extraction, structured visual representation space construction, multi-granular compositional alignment, and grounding inference. Given an image and a referring expression, a frozen YOLOv3 DarkNet-53 detector extracts top-\(K\) candidate anchor features, which are dynamically fused with foundation model representations (DINOv2 and Depth Anything v2) via learned routing weights. Concurrently, an LSTM with multi-head self-attention encodes the expression, yielding both a full-sentence embedding and a subject noun chunk embedding. The model then projects anchor features onto learnable global semantic bases to construct Unary Object Representations (UOR) and computes explicit pairwise relational features to construct Pairwise Relational Representations (PRR). These dual visual components undergo contrastive alignment against their linguistic counterparts under hierarchical consistency and basis diversity regularizations.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Input: Image & Referring Expression"] --> B["Multi-Source Encoding & Dynamic Routing<br/>YOLOv3 Anchors + Foundation Cues"]
B --> C["Unary Semantic Modeling<br/>Global Semantic Basis Projection (UOR)"]
B --> D["Explicit Relation Modeling<br/>8D Geometric & Semantic Pairwise Interaction (PRR)"]
C --> E["Compositional Weakly Supervised Alignment<br/>UOR with Subject Chunk, PRR with Full Sentence"]
D --> E
E --> F["Fused Scoring Aggregation & Detection Head Decoding"]
Key Designs¶
1. Unary Semantic Modeling: Enhancing instance discriminability via learnable global semantic bases To enrich raw anchor features with fine-grained attribute semantics without relying on manual attribute categorizations, the framework introduces \(N\) learnable global semantic basis vectors \(P = \{p_1, \dots, p_N\} \subset \mathbb{R}^d\) spanning a dataset-wide semantic subspace. For each foundation-enriched anchor \(\tilde{a}_i\), soft attention weights project the anchor onto this subspace to extract an adaptive semantic component \(a_i^{\text{una}} = \sum_{j=1}^N \gamma_{ij} p_j\). To preserve spatial localization cues from the detector alongside enriched high-level semantics, a gated integration mechanism produces the final unary object representation: $\(\hat{a}_i = g_i \odot a_i^{\text{una}} + (1 - g_i) \odot \tilde{a}_i, \quad g_i = \sigma(\text{MLP}([\tilde{a}_i \parallel a_i^{\text{una}}]))\)$ To prevent basis vector collapse and encourage diverse semantic coverage across categories, an orthogonality diversity regularizer is enforced: \(\mathcal{L}_{\text{div}} = \| B B^\top - I \|_F^2\), where \(B\) represents the \(\ell_2\)-normalized basis matrix.
2. Explicit Relation Modeling: Preserving second-order dependencies via pairwise relational units Unlike message-passing graph networks that fold contextual information back into node embeddings, this module explicitly maintains second-order relational units \(r_{ij} \in \mathbb{R}^d\) for every anchor pair \((i, j)\). An 8-dimensional geometric descriptor \(g_{ij} = [\Delta x, \Delta y, \log(w_j/w_i), \log(h_j/h_i), d_{\text{rel}}, \cos\theta, \sin\theta, \text{IoU}]\) captures relative position, scale, displacement, and orientation, and is transformed into \(\delta_{ij}\) via an MLP. Anchor semantic features and geometric descriptors are concatenated and modulated by a learnable relation confidence gate \(w_{ij}\): $\(r_{ij} = w_{ij} \cdot \text{MLP}([\tilde{a}_i \parallel \tilde{a}_j \parallel \delta_{ij}]), \quad w_{ij} = \sigma(\text{MLP}([\tilde{a}_i \parallel \tilde{a}_j \parallel \delta_{ij}]))\)$ This creates a distinct \(K \times K \times d\) pairwise relational representation space, preserving inter-object relational geometry and semantics without loss of structure.
3. Compositional Weakly Supervised Alignment: Multi-granular contrastive matching with hierarchical consistency To mirror language granularity, the referring expression is parsed into a subject noun chunk embedding \(s_c\) and a global sentence embedding \(s\). Unary alignment measures instance-level affinity \(s_i^u = \hat{a}_i^\top s_c\) and optimizes an InfoNCE loss \(\mathcal{L}_{\text{una}}\) over candidate anchors with respect to pseudo-positive \(i^*\). Pairwise alignment computes relational compatibility \(s_{ij}^r = r_{ij}^\top s\) and optimizes \(\mathcal{L}_{\text{pair}}\) across all \(K^2\) pairs. To prevent multi-granular alignments from drifting apart without box supervision, a hierarchical consistency loss \(\mathcal{L}_{\text{cons}}\) compels pairwise relational scores to concentrate on anchors prioritized by unary matching: $\(\mathcal{L}_{\text{cons}} = -\log \sum_{i \in \text{TopK}(v_{\text{una}})} v_{\text{pair}}(i)\)$ ensuring mutual agreement and self-supervised refinement between object-level candidates and global relations.
Loss & Training¶
The overall training objective is formulated as the joint combination of cross-modal contrastive losses and structural regularizations: $\(\mathcal{L} = \mathcal{L}_{\text{una}} + \mathcal{L}_{\text{pair}} + \mathcal{L}_{\text{div}} + \mathcal{L}_{\text{cons}}\)$ During inference, candidate anchor \(i\) receives a bidirectional symmetric relational score \(\text{Sim}_i^{\text{pair}} = \frac{1}{2} [\sum_j \exp(r_{ij}^\top s) + \sum_j \exp(r_{ji}^\top s)]\) and a unary score \(\text{Sim}_i^{\text{una}} = \hat{a}_i^\top s_c\). The final grounding confidence is obtained via linear fusion: $\(\text{Score}_i = \lambda \cdot \text{Sim}_i^{\text{pair}} + (1 - \lambda) \cdot \text{Sim}_i^{\text{una}}\)$ with \(\lambda = 0.6\) across all datasets. The bounding box of the highest-scoring anchor is decoded by the detection head as the output prediction.
Key Experimental Results¶
Main Results¶
Evaluation across RefCOCO, RefCOCO+, and RefCOCOg benchmarks using the standard [email protected] accuracy metric demonstrates clear superiority over prior weakly supervised approaches:
| Dataset | Split | SVCR (Ours, YOLOv3) | DViN (CVPR 2025) | WeakMCN (CVPR 2025) | RefCLIP (CVPR 2023) | Gain over DViN |
|---|---|---|---|---|---|---|
| RefCOCO | val | 71.06 | 67.67 | 69.20 | 60.36 | +3.39% |
| RefCOCO | testA | 72.60 | 70.90 | 69.88 | 58.58 | +1.70% |
| RefCOCO | testB | 65.55 | 59.39 | 62.63 | 57.13 | +6.16% |
| RefCOCO+ | val | 54.17 | 52.54 | 51.90 | 40.39 | +1.63% |
| RefCOCO+ | testA | 60.22 | 57.52 | 57.33 | 40.45 | +2.70% |
| RefCOCO+ | testB | 44.63 | 45.31 | 43.10 | 38.86 | -0.68% |
| RefCOCOg | val | 56.99 | 55.04 | 54.62 | 47.87 | +1.95% |
Under a modern YOLOv5 backbone, SVCR* further pushes performance to 74.51% (val), 76.93% (testA), and 70.46% (testB) on RefCOCO, and 60.40% on RefCOCOg val, substantially closing the gap to fully supervised models.
Ablation Study¶
Ablation on the DINOv2 + Depth Anything v2 baseline isolates the exact contribution of each component:
| Config | Alignment | RefCOCO val | RefCOCO testB | RefCOCO+ val | RefCOCOg val | Note |
|---|---|---|---|---|---|---|
| Baseline (DINO + Depth) | Flat (FA) | 66.59 | 59.94 | 51.22 | 51.08 | Standard anchor matching with global text |
| + UOR | Flat (FA) | 69.93 | 64.30 | 53.50 | 53.31 | Unary basis projection boosts object-level attributes |
| + PRR | Compositional (CA) | 69.22 | 63.47 | 52.45 | 54.31 | Explicit pairwise modeling strengthens complex relations |
| + UOR + PRR (SVCR Full) | Compositional (CA) | 71.06 | 65.55 | 54.17 | 56.99 | Dual structured representations yield a massive +5.91% gain on RefCOCOg |
Furthermore, comparing relational structures reveals that re-aggregating relations into unary anchors (+ UOR + URR) achieves only 55.11% on RefCOCOg val, whereas retaining explicit pairwise representations (+ UOR + PRR) reaches 56.99%, confirming the critical necessity of decoupled second-order units.
Key Findings¶
- Distinct functional complementarity of UOR and PRR: UOR provides the largest boost on object-centric, attribute-focused short queries (e.g., +4.36% on RefCOCO testB), while explicit PRR dominates on long, relation-heavy expressions (yielding +3.23% alone and +5.91% in combination on RefCOCOg val).
- Geometric depth priors outperform vision-language features: Combining DINOv2 with Depth Anything v2 consistently surpasses DINOv2 + CLIP (e.g., 65.55% vs. 60.43% on RefCOCO testB), demonstrating that continuous 3D geometric hints synergize much better with explicit relational modeling than flat visual-language embeddings.
- Basis size capacity saturation: Performance scales steadily with semantic basis count up to \(N = 100\), beyond which redundancy and weak supervision noise slightly degrade generalization.
Highlights & Insights¶
- From implicit context absorption to explicit second-order representation: Challenges the entrenched paradigm of compressing all relational context into single-anchor vectors. It proves that maintaining an explicit \(K \times K\) relation space under weak supervision is not only tractable but fundamentally solves feature entanglement.
- Hierarchical consistency as a self-supervised stabilizer: Elegantly bridges local noun chunk grounding with global relational inference, guiding pseudo-positive selection without manual annotations.
- Broad applicability across structured grounding tasks: The decoupled formulation (unary attributes + pairwise geometry/semantics + multi-granular contrast) provides a versatile blueprint for weakly supervised spatio-temporal action grounding, audio-visual localization, and open-vocabulary scene graph understanding.
Limitations & Future Work¶
- Vulnerability to negation and exclusion logic: Failure analysis shows frequent misgrounding when handling negative expressions (e.g., "boy not wearing red"), where the target must be identified by rejecting candidate attributes.
- Quadratic computational complexity with anchor count: Maintaining an explicit \(K \times K\) matrix scales quadratically, which could become prohibitive for dense anchor grids (\(K > 100\)) without sparse pruning strategies.
- Confusion under severe occlusion and visual ambiguity: Performance drops in heavily crowded scenes where geometric descriptors fail to sufficiently discriminate partially overlapping candidates.
Related Work & Insights¶
- vs APL (ECCV 2024): APL introduces prompt tuning with handcrafted geometric and category heuristics but still forces alignment into a flat unary space. SVCR constructs adaptive continuous bases and explicitly models pairwise relations.
- vs DViN (CVPR 2025): DViN enriches features via dynamic routing across foundation models but retains conventional flat anchor-text matching. SVCR leverages foundation cues as base inputs and innovates on structured multi-granular composition.
- vs Supervised Graph Grounding (e.g., MattNet / VG-GMN): Supervised graph networks rely on ground-truth bounding boxes and off-the-shelf linguistic parsers to supervise edges, whereas SVCR successfully learns structured visual composition entirely from weak image-text pairs.
Rating¶
- Novelty: ⭐⭐⭐⭐☆ Moves beyond flat anchor matching by decoupling unary and pairwise relational spaces under weak supervision.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive evaluations across three benchmarks, rigorous ablations on basis size, losses, and representation forms.
- Writing Quality: ⭐⭐⭐⭐⭐ Well-formulated problem motivation, crystal-clear mathematical derivations, and illuminating qualitative analysis.
- Value: ⭐⭐⭐⭐☆ Sets a compelling new benchmark and conceptual direction for weakly supervised visual-linguistic grounding.