Revisiting Scene Graph Generation from the Perspective of Detector-Conditioned Reachability

Conference: ECCV 2026
Paper: ECCV Official Link
Code: Dual-SGG
Area: Object Detection
Keywords: Scene Graph Generation, Visual Relationship Detection, Reachability Analysis, Dual-Query Mechanism, Transformer

TL;DR

This paper quantitatively analyzes prediction discrepancies between detector-based and query-based SGG methods through the lens of detector-conditioned reachability, and proposes Dual-SGG—a unified framework combining top-down and bottom-up triplet queries within a single decoder to achieve state-of-the-art relationship detection without complex post-hoc calibration.

Background & Motivation

Scene graph generation (SGG) aims to parse an image into a structured graph where entities serve as nodes and pairwise relationships form directed edges, typically formatted as (subject, predicate, object) triplets. Existing SGG frameworks are predominantly bifurcated into detector-based and query-based approaches according to their underlying reasoning paradigms. Detector-based approaches rely on an upstream object detector to identify entities and subsequently predict predicates over candidate pairs. However, these models inherently suffer from what the authors term the "detector constraint": if either the subject or object instance is omitted or mislocalized by the object detector, the relationship cannot be recovered. In contrast, query-based methods adopt the DETR-style philosophy, updating learnable triplet queries to predict relationships end-to-end without explicit entity pair enumeration.

For years, the research community has evaluated both paradigms using aggregate benchmarks such as Recall@K and mean-Recall@K. While these metrics summarize overall accuracy, they obscure the distinct predictive behaviors resulting from the different reasoning mechanisms. Specifically, it has remained unproven whether query-based models genuinely recover the triplets missed by detector-based methods due to detector constraints, or how their reachability distributions diverge in practice.

To resolve this ambiguity, the authors introduce a controlled experimental protocol on the Visual Genome dataset by partitioning ground-truth triplets into two disjoint subsets: reachable triplets (Det-T), where both subject and object are successfully detected by a pre-trained detector, and unreachable triplets (UDet-T), where at least one entity is undetected. Evaluating both model types on these subsets reveals clear empirical complementarity: detector-based models excel on Det-T but completely collapse on UDet-T, whereas query-based models capture significantly more UDet-T triplets at the expense of substantial drops on Det-T. The core idea is to consolidate detector-based conditioning and query-based global exploration into a unified transformer decoder via a dual-query design, thereby capturing the complementary strengths of both mechanisms without requiring ad-hoc score ensembling.

Method

Overall Architecture

Dual-SGG unites the precise localization priors of detector-based methods with the unconstrained exploration capability of query-based models. The overall pipeline comprises an object detector (Deformable-DETR), an Entity Pair Selector (EPS), and a unified triplet decoder. First, the detector extracts image features and proposes object candidates. The EPS ranks these candidates across both content and spatial dimensions, selecting the Top \(N_{td}\) pairs to construct top-down triplet queries (TD-Qs). Concurrently, a set of \(N_{bu}\) bottom-up triplet queries (BU-Qs) with center-biased anchors are initialized to perform global search independently of detector proposals. Both query sets are fed into a stacked transformer decoder equipped with a self-attention mask to prevent prior leakage, followed by multi-task triplet detection heads and non-maximum suppression (NMS) to generate the final prediction set.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Image Input and Feature Extraction<br/>Deformable-DETR Backbone & Encoder-Decoder"] --> B["1. Entity Pair Selector (EPS): Mitigating Combinatorial Explosion via Dual Compatibility"]
    A --> C["2. Dual-Query Initialization: Decoupling Top-Down Conditioning and Bottom-Up Exploration"]
    B -->|Select Top-Ntd Entity Pairs| C
    C --> D["3. Masked Self-Attention and Joint Triplet Decoding: Preventing Prior Leakage"]
    D --> E["Triplet Detection Heads & NMS<br/>Output Unified Final Predictions"]

Key Designs

1. Entity Pair Selector (EPS): Mitigating Combinatorial Explosion via Dual Compatibility Directly considering all pairwise combinations of \(N_e\) detected entities yields \(N_e^2\) candidate pairs, which imposes prohibitive computational complexity on subsequent transformer layers. Rather than using an intricate post-hoc re-weighting network, EPS serves as a lightweight, fast filter. Using self-attention and a feed-forward network over entity features, EPS computes content logits \(S^{ct}\) and spatial logits \(S^{sp}\). Content logits evaluate semantic affinity via dot-product projections to select candidate pairs corresponding to ground-truth triplets. Meanwhile, spatial logits measure bounding-box topological overlap to prioritize informative hard negative pairs that resemble positive configurations. During inference, Top \(N_{td}\) pairs are retained based strictly on content scores, drastically reducing the search space while preserving high-recall entity candidates.

2. Dual-Query Initialization: Decoupling Top-Down Conditioning and Bottom-Up Exploration To simultaneously capture detector-conditioned relationships and global image context, Dual-SGG introduces two distinct query streams. Top-down queries (TD-Qs) initialize their content embeddings and triplet anchors by concatenating the features and bounding boxes of the entity pairs selected by EPS. These queries operate strictly within the detector-conditioned reachability domain. In contrast, bottom-up queries (BU-Qs) initialize content vectors with learnable parameters, while their spatial anchors are initialized as pairwise frozen boxes placed at the image center. This center-biased initialization guides the decoder to focus initially on the image center and gradually expand outward across successive layers, effectively discovering triplets that lie beyond the detector's reachability boundary.

3. Masked Self-Attention and Joint Triplet Decoding: Preventing Prior Leakage Jointly optimizing TD-Qs and BU-Qs in an unconstrained transformer decoder poses a major risk: the rich entity priors embedded in TD-Qs quickly contaminate the BU-Qs through unconstrained self-attention, eroding their capacity for independent global exploration. Dual-SGG enforces a unidirectional self-attention mask within each decoder layer that permits attention from BU-Qs to TD-Qs but blocks the reverse information flow from TD-Qs into BU-Qs. This maintains the exploratory autonomy of BU-Qs while allowing both query streams to share unified feature representations and score scales, eliminating the need for post-hoc probability calibration across separate models.

A Worked Example

Consider an input image showing a cluttered dining room where the detector correctly localizes a dining table, chairs, and a wine glass, but fails to identify a small spoon resting inside an obscured bowl due to extreme occlusions: 1. Candidate Proposal & EPS Selection: The detector outputs bounding boxes and features for the table, chairs, and glass. EPS computes pairwise scores and routes the high-confidence (glass, on, table) pair into the TD-Qs stream with initialized box coordinates. 2. Dual-Query Reasoning with Attention Mask: Within the transformer decoder, TD-Qs rapidly converge on the localized pair and accurately predict (glass, on, table). Simultaneously, the center-biased BU-Qs expand outward from the image center; unencumbered by detector misdetections, they register the subtle visual cues of the spoon handle and bowl rim, decoding (spoon, in, bowl). The attention mask ensures TD-Qs do not dominate the representations of the BU-Qs. 3. Unified Output via NMS: The predictions from both branches are pooled into a single set and filtered with NMS, seamlessly recovering the obscured relation without separate score normalization.

Loss & Training

The overall training objective is formulated as a multi-task loss optimized end-to-end: $\(\mathcal{L} = \mathcal{L}^{e} + \mathcal{L}^{eps} + \mathcal{L}^{TD} + \mathcal{L}^{BU}\)$ Here, \(\mathcal{L}^e\) trains the Deformable-DETR detector via Focal loss and L1/GIoU bounding-box regression. \(\mathcal{L}^{eps}\) comprises spatial loss \(\mathcal{L}^{sp}\) and content loss \(\mathcal{L}^{ct}\) to supervise candidate pair selection. \(\mathcal{L}^{TD}\) directly penalizes subject, object, and predicate errors against aligned detector ground-truth labels for TD-Qs. \(\mathcal{L}^{BU}\) applies standard one-to-one Hungarian matching (omitting predicate cost for computational efficiency) between predictions and ground-truth triplets. All newly introduced loss terms are assigned unit weights, eliminating manual hyperparameter balancing.

Key Experimental Results

Main Results

On the Visual Genome test set under the standard Graph-Constraint protocol, Dual-SGG outperforms prior ResNet-50-based methods across all metrics while operating at a practical inference speed of 14.0 FPS. When combined with Logit Adjustment (LA) for unbiased scene graph generation, Dual-SGG achieves state-of-the-art performance.

Model	Backbone	FPS	R@50	R@100	mR@50	mR@100	F@50	F@100
RelTR (TPAMI'22)	ResNet-50	13.4	27.5	30.7	10.8	12.3	15.5	17.6
Relationformer (ECCV'22)	ResNet-50	8.5	28.4	31.3	9.3	10.7	14.0	15.9
ISG (NeurIPS'22)	ResNet-50	6.0	29.7	32.1	8.0	8.8	12.6	13.8
EGTR (CVPR'24)	ResNet-50	14.7	30.2	34.3	7.9	10.1	12.5	15.6
Dual-SGG (Ours)	ResNet-50	14.0	33.5	38.5	10.3	12.3	15.8	18.6
EGTR+LA (CVPR'24)	ResNet-50	14.7	24.2	26.7	17.1	21.4	20.0	23.8
Hydra-SGG (ICLR'25)	ResNet-50	5.3	28.4	33.1	16.0	19.7	20.5	24.7
Salience-SGG (WACV'26)	ResNet-50	13.3	28.8	33.4	18.0	21.6	22.1	26.2
Dual-SGG+LA (\(\tau=0.2\), Ours)	ResNet-50	14.0	30.2	35.2	18.7	22.2	23.1	27.3
Dual-SGG+LA (\(\tau=0.3\), Ours)	ResNet-50	14.0	25.3	30.0	22.2	25.2	23.6	27.4

Ablation Study

Ablations on query groups under a constant total query budget (\(N_{td}+N_{bu}=1100\)) on the VG test set (Det-T: 98,103 triplets; UDet-T: 54,123 triplets) validate the mutual indispensability of TD-Qs and BU-Qs.

Configuration	R@50/100	mR@50/100	micro-R@50/100	micro-DR@50/100	micro-UDR@50/100	Note
Detector-based Baseline	31.6 / 36.0	9.1 / 11.2	30.2 / 34.8	46.7 / 53.8	0.2 / 0.3	Collapses entirely on unreachable set UDet-T
Dual-SGG w/o BU-Qs	31.9 / 36.3	8.6 / 10.3	30.4 / 35.2	46.3 / 53.4	1.5 / 2.2	Restricted by detector reachability boundaries
Dual-SGG w/o TD-Qs	31.3 / 34.7	10.0 / 11.6	30.0 / 33.7	44.1 / 49.0	4.5 / 5.9	High UDet-T recall but sacrifices Det-T
Dual-SGG (Full)	33.5 / 38.5	10.3 / 12.3	31.8 / 36.9	47.3 / 54.3	3.7 / 5.2	Maintains top Det-T recall while gaining UDet-T

Ablations on architectural components further demonstrate: - Discarding spatial score \(S^{sp}\) or content score \(S^{ct}\) in EPS reduces R@50 to 31.5 and 33.0 respectively, showing their utility in identifying hard negatives and positive priors. - Removing masked self-attention (MSA) drops R@50 to 32.8, verifying that unconstrained leakage from TD-Qs degrades BU-Qs' exploration. - Replacing center-biased anchor initialization with uniform DETR grid boxes drops R@50 to 32.9.

Key Findings

• Superiority over Post-Hoc Fusion: Merging independently trained detector-based and query-based models using Max, Average, or Normalization heuristics yields at best 32.2 R@50, substantially underperforming the unified Dual-SGG (33.5 R@50). Joint decoding avoids calibration pitfalls.
• Robustness Under Stronger Detectors: When upgrading the detector backbone to Swin-B with 900 entity queries, the unreachable subset UDet-T shrinks from 35% to 28%. Even so, Dual-SGG still achieves +1.7 R@50 over Dual-SGG w/o BU-Qs (38.8 vs 37.1), confirming that detector bottlenecks persist across model scales.

Highlights & Insights

• Reachability Diagnosis Paradigm: Decomposing ground-truth relationships into Det-T and UDet-T clarifies previously puzzling performance discrepancies between detector-based and query-based SGG architectures.
• Masked Dual-Query Transformer: Unidirectional attention masking enables two fundamentally different reasoning paradigms to share a single transformer decoder without representational collapse.
• Center-Biased Anchor Scheduling: Initializing exploratory anchors at the center and allowing spatial attention to expand outward provides an effective spatial prior for unconditioned relationship discovery.

Limitations & Future Work

• Residual Recall Losses under Complex Occlusion: Qualitative analysis indicates that in rare, highly crowded images, Dual-SGG fails to simultaneously retain every ground-truth triplet discovered by the individual specialized models.
• Fixed Query Capacity in EPS: Selecting a fixed number of Top \(N_{td}\) pairs can drop valid relationships in scenes with exceptionally dense visual interactions.
• Future Directions: Exploring dynamic query allocation and graph-aware non-maximum suppression are promising avenues to enhance triplet filtering.

Related Work & Insights

• vs. RelTR / ISG (Query-based): RelTR and ISG avoid entity pair enumeration but sacrifice localized entity conditioning, resulting in lower R@50 (27.5 and 29.7); Dual-SGG restores strong localization via TD-Qs, boosting R@50 to 33.5.
• vs. EGTR (Detector-based): EGTR processes all entity pairs with an MLP, suffering from combinatorial compute and zero recall on undetected entities; Dual-SGG prunes the pair space with EPS and recovers unseen triplets with BU-Qs.
• vs. DINO / DN-DETR (Auxiliary Queries): Auxiliary queries in DETR object detection are strictly training-time denoising targets discarded during inference; in Dual-SGG, both query streams operate actively during both training and inference.

Rating

• Novelty: ⭐⭐⭐⭐⭐ Introduces detector-conditioned reachability analysis and an elegant dual-query architecture to fuse complementary predictive behaviors.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive evaluations across Visual Genome, GQA-200, and Open Images v6 with meticulous sub-population dissections.
• Writing Quality: ⭐⭐⭐⭐⭐ Crisp problem formulation, coherent experimental motivation, and intuitive visualizations.
• Value: ⭐⭐⭐⭐⭐ Clarifies foundational evaluation blind spots in SGG and establishes a strong, unified design principle for visual relation learning.