HSGG: Training-Free Hierarchical Scene Graph Generation with Geometry-Guided Relation Reasoning¶

Conference: ICML 2026
Paper: OpenReview
Area: Object Detection / Scene Understanding
Keywords: scene graph generation, hierarchical entity perception, geometric filtering, contrastive decoding, relational hallucination

TL;DR¶

HSGG discovers entities from wholes to parts, propagates attributes from parts to wholes, and then combines geometric filtering with visually grounded predictions contrasted against geometry-induced hallucination priors, achieving 15.1 zR@100 on VG150 and 14.1 on PSG under SGDet without task-specific training (original Table 1).

Background & Motivation¶

Scene graph generation must do more than recognize a person and a bicycle: it must identify who is doing what to whom. Conventional approaches depend on dense object-box and relation-triplet annotations, while predicate distributions are strongly long-tailed. Frequent predicates such as on and has are easier to learn than rare combinations and fine-grained actions. Pretrained vision-language models (VLMs) enable direct relation queries, but familiar linguistic co-occurrence is not evidence about the current image. A model may predict riding for a person merely standing beside a bicycle; correct object categories do not guarantee a correct relation.

Another problem occurs earlier. If the entity set contains only a few salient whole objects, even accurate relation decoding cannot answer questions about gloves, masks, face shields, or clothing patterns. A flat object list loses the organization of wholes and parts, while generating one long description can still miss small instances. HSGG therefore separates entity coverage from relation reliability: it recursively proposes and localizes related details, then feeds their attributes back into whole-object descriptions. With more nodes available, it must also control the number of pairwise relation queries so that richer representations do not simply produce more spurious edges.

Geometry serves two distinct purposes. It can reject candidates with implausible image-plane or depth separation, but cannot by itself establish semantic relations such as holding or wearing. Conversely, the mistakes a model makes from categories and geometry alone can serve as a prior to suppress during decoding. Core Idea: expand the hierarchical entity set, use geometry to select which pairs deserve a query, and then strengthen visual evidence relative to geometry- and language-induced hallucinations instead of treating spatial proximity as proof of a relation.

Method¶

Overall Architecture¶

The input is an image and the output is a hierarchical scene graph. Nodes contain categories, bounding boxes, and attributes; semantic edges represent directional subject-predicate-object triplets. BHEP constructs a multilevel entity representation. The relation stage then runs GA-RF followed by GG-CRD, first reducing the candidate set and then decoding a predicate for each retained pair. The pipeline performs inference with pretrained components rather than fitting a new scene graph model.

Hierarchy and semantic relations are different structures. A person can occupy a higher level than clothing, gloves, and finer details. In-layer relations connect entities at comparable abstraction levels, such as beside between two people; inter-layer relations can connect a person to clothing through wearing. Parent-child organization provides candidates and attribute context, but does not automatically imply a fixed predicate. Algorithm 1 filters pairs from the node set rather than restricting inference to same-layer neighbors or existing parent-child edges.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    input["Input image"] --> bhep["BHEP<br/>Bidirectional Hierarchical<br/>Entity Perception"]
    bhep -->|Entity boxes, categories, attributes| garf["GA-RF<br/>Geometry-Aware<br/>Relation Filtering"]
    input --> depth["Pretrained depth estimation"]
    depth -->|Median depths| garf
    garf -->|Retained object pairs| ggcrd["GG-CRD<br/>Geometry-Grounded<br/>Contrastive Relation Decoding"]
    input -->|Visual crop for the expert| ggcrd
    ggcrd --> output["Hierarchical scene graph<br/>In-layer and inter-layer relations"]

Key Designs¶

1. BHEP: Bidirectional Hierarchical Entity Perception

The top-down pass first asks the VLM to identify independent whole objects and then proposes child entities conditioned on parent categories. The root prompt in Appendix E.1 scans foreground, middle ground, and background while initially excluding dependent parts. The child prompt subsequently proposes at most five accessories, components, or decorations per object category, avoids synonyms and repeated categories, and prioritizes clothing and accessories for people. An open-vocabulary detector localizes each candidate to form a category-box pair, rather than treating a textual proposal as an already verified visual instance.

The hierarchy is closer to entity-centered semantic decomposition than a strictly anatomical part-of tree. The child prompt permits physical attachment, decorative association, and functional association, including tires and cars, patterns and clothing, or remote controls and televisions. The open-vocabulary localizer then checks which image instances correspond to these category proposals.

The bottom-up pass infers attributes rather than proposing another set of objects. It first reads leaf-node image regions to describe color, material, state, and other details. These descriptions become textual context for the parent, alongside the parent's own visual region. Appendix E.2 asks for a phrase of no more than 20 words describing the red-box target and says to use child context only when helpful. Face-shield details can thus enrich a helmet description, while helmet and clothing details help distinguish people, instead of relying on a generic whole-image paragraph for every distinction.

The directions have complementary roles: downward expansion increases the set of localizable entities, while upward reasoning enriches existing node semantics. Parent descriptions are not necessarily concatenations of child attributes, and there is no new graph neural network loss. A pretrained VLM performs the fusion under image and text conditioning. The default maximum hierarchy depth is 3. Appendix C.2 notes that deeper expansion increasingly introduces overlapping and redundant entities, so more levels are not automatically better.

2. GA-RF: Geometry-Aware Relation Filtering

As the entity set grows, the number of candidate relations grows quadratically. Image-plane center distance alone can incorrectly remove interactions involving large objects, while box overlap can conflate objects at different depths. GA-RF first combines normalized image-plane distance and depth separation, then applies a scale correction. Following Equations (3) through (5), the first step is:

\[ \begin{aligned} \delta_{ij} &= \|m_i-m_j\|_2,\\ d_{\mathrm{raw}} &= \frac{\delta_{ij}}{\sqrt{H^2+W^2}} +\frac{|z_i-z_j|}{\Delta Z}. \end{aligned} \]

Here \(i,j\) identify the two objects, \(m_i,m_j\) are their 2D box centers, and \(\delta_{ij}\) is the center separation. \(H,W\) are image height and width, \(z_i,z_j\) are median depths over the respective boxes, and \(\Delta Z\) is the scene depth range. The raw score \(d_{\mathrm{raw}}\) sums two separately normalized terms instead of directly mixing pixels and depth values.

The second step introduces box diagonal lengths \(s_i,s_j\) to compute the scale factor \(\psi_{ij}\) and final filtering score \(d_{\mathrm{final}}\):

\[ \begin{aligned} \psi_{ij} &= \frac{\delta_{ij}}{\delta_{ij}+\tfrac12(s_i+s_j)},\\ d_{\mathrm{final}} &= d_{\mathrm{raw}}\psi_{ij}. \end{aligned} \]

At a fixed center separation, larger boxes produce a smaller scale factor and therefore a weaker penalty.

Only pairs satisfying \(d_{\mathrm{final}}<\tau\) are retained, with \(\tau=0.3\) in all experiments. This step outputs a sparse candidate set, not predicates: proximity cannot establish riding or holding. The score is symmetric under swapping the objects, whereas predicates are directional, so filtering cannot determine subject-object direction either. It reduces expensive downstream VLM decoding calls, but does not remove the quadratic enumeration of pair checks in Algorithm 1.

3. GG-CRD: Geometry-Grounded Contrastive Relation Decoding

Filtering determines which pairs to query, but cannot prevent the VLM from predicting a linguistically familiar yet visually unsupported predicate. GG-CRD assigns two roles to the same core VLM under different inputs and prompts. The expert sees a crop containing the object pair. Its prompt identifies subject and object with differently colored boxes and explicitly specifies the direction from subject to object. It should return no relation when no clear relation is visible. The expert is therefore not merely a language classifier receiving category names.

The hallucination-prior branch (called the amateur in the paper) receives no image, only textual information such as object categories, bounding boxes, and depths. Appendix E.3 explicitly asks it for the incorrect relationship that a model is most likely to hallucinate without seeing the image. Examples include mistaking a nearby person and bicycle for riding, and a dog and ball for playing with. It reuses the same VLM as the expert, deliberately eliciting bias through its prompt to provide a contrastive signal to subtract during decoding.

At the same generation step \(t\), both branches produce next-token logits conditioned on the same generated prefix \(y_{<t}\). The expert and hallucination-prior logits are denoted \(L_{\mathrm{ex}}\) and \(L_{\mathrm{am}}\), respectively. The expert probability distribution \(P_{\mathrm{ex}}\) first restricts the available tokens so that differences between two extremely low-confidence predictions cannot elevate an implausible answer. Let \(V\) be the vocabulary and \(y,w\) candidate tokens in it. The adaptive plausibility set in Equation (8) is:

\[ P_{\mathrm{ex}}=\operatorname{softmax}(L_{\mathrm{ex}}),\qquad V_{\mathrm{apc}}=\{y\in V:P_{\mathrm{ex}}(y)\geq\beta\max_{w\in V}P_{\mathrm{ex}}(w)\}. \]

Within this set, Equation (9) adjusts the logits. Tokens outside it receive negative infinity, and a softmax distribution determines the next token:

\[ \widetilde L(y)= \begin{cases} L_{\mathrm{ex}}(y)+\alpha\bigl(L_{\mathrm{ex}}(y)-L_{\mathrm{am}}(y)\bigr),&y\in V_{\mathrm{apc}},\\ -\infty,&y\notin V_{\mathrm{apc}}, \end{cases} \qquad y_t\sim\operatorname{softmax}(\widetilde L). \]

The defaults are \(\alpha=0.1\) and \(\beta=0.1\). The former controls bias suppression relative to the original expert logits; the latter requires an expert token probability of at least one tenth of the expert's maximum probability. It does not retain 10% of the vocabulary and is not an object detection confidence threshold. Contrast occurs at each token decoding step, rather than having a second model score completed triplets afterward.

The prompts require only a predicate or no relation, not a full triplet. Decoded semantic predicates are combined with the localized subject and object to form edges, while no relation expresses abstention.

A Worked Example¶

Suppose a person is standing beside a bicycle, not riding it. BHEP has localized both entities, and assume that this pair passes GA-RF. Focus now on how GG-CRD corrects the common-knowledge error riding. The following logits are invented teaching values, not paper experiments or measured model outputs. For hand calculation, assume a vocabulary of just three single-token candidates, abbreviated as riding, beside, and no_relation in the table. Real predicates and no relation may span multiple tokens; this simplified vocabulary is not the actual tokenizer output.

Fix the same generation step and generated prefix: the image-conditioned expert still ranks riding slightly above beside, while the image-free hallucination-prior branch favors riding much more strongly. With the paper's defaults \(\alpha=0.1\) and \(\beta=0.1\), compute \(\widetilde L=1.1L_{\mathrm{ex}}-0.1L_{\mathrm{am}}\) for each token:

Teaching candidate token	Expert logit \(L_{\mathrm{ex}}\)	Hallucination-prior logit \(L_{\mathrm{am}}\)	Adjusted logit \(\widetilde L\)
`riding`	2.0	4.0	1.80
`beside`	1.9	0.0	2.09
`no_relation`	0.0	0.0	0.00

The plausibility set must also be checked before contrast. A softmax probability ratio equals the exponential of the logit difference, so the ratios to the maximum expert probability are \(1\), \(e^{-0.1}\approx0.9048\), and \(e^{-2}\approx0.1353\). All meet \(\beta=0.1\) and remain eligible. The adjusted score for riding is then \(2.0+0.1(2.0-4.0)=1.80\), while beside receives \(1.9+0.1(1.9-0.0)=2.09\): the top-ranked candidate changes from riding to beside.

Softmax preserves this ranking. Taking the maximum would select beside at this step, whereas sampling as in the original equation does not guarantee it. The mechanism penalizes tokens especially favored without the image at the same step; it does not subtract probabilities of already completed relations. Actual decoding updates the prefix and repeats this process until the predicate is complete.

Loss & Training¶

HSGG introduces no task-specific training loss. Appendix A names LLaVA-OneVision-1.5-8B as the core VLM, LLMDet as the open-vocabulary localizer, and Depth Anything V3 as the depth component; these names are preserved as written. The default hierarchy depth is 3, with \(\tau=0.3\), \(\alpha=0.1\), and \(\beta=0.1\). The main operations are iterative prompting, localization, depth estimation, and dual-branch decoding.

Training-free means that assembling this SGG pipeline requires no additional task-specific parameter training. For evaluation, Section 4.1 uses Qwen3-Embedding and cosine similarity to align open-vocabulary predictions with benchmark labels; unseen is defined by the respective evaluation split.

Key Experimental Results¶

Main Results¶

The paper states that it follows standard SGDet: entities and relations are predicted from the image, rather than classifying predicates over ground-truth entities. R@K measures recall of annotated relations among the top K predicted triplets; zR@K restricts evaluation to unseen triplets. The zero-shot setting primarily tests novel combinations of known categories, whereas open-world evaluation additionally tests unseen object and predicate categories. These are different generalization claims.

Original Table 1: zero-shot SGDet, with values preserved from the table. VG150 uses 150 object categories and 50 predicates; PSG contains 133 object categories and 56 predicates. The selected comparisons below should not be treated as controlled same-backbone ablations across training regimes.

Dataset	Method	zR@20	zR@50	zR@100
VG150	OwSGG	0.7	1.2	2.0
VG150	CAPSGG	9.3	12.5	13.3
VG150	HSGG	10.1	13.5	15.1
PSG	OwSGG	2.8	4.7	6.4
PSG	PGSG	-	6.8	8.9
PSG	HSGG	7.2	11.4	14.1

Original Table 2: open-world SGDet on VG. Appendix B.1 says this setting follows the OwSGG evaluation strategy; the unseen-category setting does not remove the output-label alignment described above.

Method	R@20	R@50	R@100
OwSGG(Qwen)	-	1.6	2.4
OwSGG(LLaVA-Next)	-	1.9	2.6
OvSGTR (Swin-B)	-	6.0	10.0
HSGG	6.4	9.3	11.9

Subtracting the table entries, HSGG exceeds CAPSGG by 1.0/1.8 percentage points in VG150 zR@50/100 and OvSGTR (Swin-B) by 3.3/1.9 points in open-world R@50/100. These are the paper's reported results, not reproductions conducted for this note, and absolute recall still leaves substantial room for improvement.

Ablation Study¶

Original Table 5: component ablation on zero-shot PSG. Checkmarks are transcribed as Yes/No to avoid misreading the first row as BHEP-only. The first row enables only GG-CRD; the second enables GA-RF and GG-CRD.

BHEP	GA-RF	GG-CRD	zR@20	zR@50	zR@100
No	No	Yes	5.23	6.89	7.35
No	Yes	Yes	5.39	7.12	7.58
Yes	No	No	6.53	9.82	11.63
Yes	Yes	No	6.92	10.73	12.13
Yes	Yes	Yes	7.18	11.42	14.05

Table 1 reports the full PSG results to one decimal place and Table 5 to two; both presentations are preserved. The discrepancy between the prose gain and table entries is collected under Limitations & Future Work.

Key Findings¶

Entity coverage is a prerequisite: GG-CRD alone gives 7.35 zR@100, rising only to 7.58 with GA-RF, while BHEP-only reaches 11.63. However, the former and latter are not a single-factor comparison, so their difference is not a controlled estimate of BHEP's isolated contribution.
Adding GA-RF to BHEP moves zR@50/100 from 9.82/11.63 to 10.73/12.13; adding GG-CRD then reaches 11.42/14.05. This progression supports complementary contributions from entity discovery, candidate control, and decoding correction.
Appendix C.2 uses a GPT-4o silver standard: increasing the threshold from 0.1 to 0.5 raises candidate recall from 53.12% to 79.69% while reducing precision from 24.46% to 8.81%. Recall is 76.56% at 0.3. These are not SGDet zR@K results or human-verified relation accuracies.
Downstream evaluations provide supporting evidence: in original Table 3, LLaVA-OV-1.5-8B improves from 76.96% to 79.58% overall on V*Bench. Table 4 reports unified scores of 59.70 for image input and 62.97 for HSG-only input. The latter is a 3.27-point increase on a 0-to-100 scale, not an accuracy under a shared metric; the graph still comes from upstream image processing, so the full system is not vision-free.

Highlights & Insights¶

Explicitly modeling what common knowledge gets wrong creates a reference distribution more closely matched to relational errors than generic input noise. The transferable idea is to design contrastive conditions around a specific error mechanism, not to penalize all priors indiscriminately.
Structured discovery and reliable decoding depend on each other. Hallucination suppression cannot recover a small entity that never entered the candidate set; expanding entities before correcting edges follows the task's error propagation more closely.
Geometry plays different roles in the two stages: it retains plausible candidates during filtering and exposes geometry-only biases during decoding. This separates the claim that geometry is useful from the stronger, unsupported claim that it is sufficient.

Limitations & Future Work¶

Appendix F acknowledges dependence on VLM and depth-estimation quality and proposes lightweight geometric adapters, distillation, and video scene graphs. Although the paper describes computational cost as comparable to existing methods, the material read here provides no end-to-end latency table sufficient for an independent comparison. Training-free does not imply inexpensive inference.
Hierarchical candidates depend on common-knowledge prompts: the main method describes image-conditioned parent expansion, while the explicit input slot in the appendix's child prompt is primarily a category list. Localization errors, duplicate instances, and associative parent-child structures affect later reasoning. Contrastive decoding can also suppress common but correct relations, motivating false-removal analysis.
GA-RF is a heuristic filter, not a 3D contact test. With a nonzero depth range, nondegenerate boxes with coincident centers have zero scale factor and final score, retaining the pair even with a large depth difference. The equations do not specify numerical handling of a zero depth range. Algorithm 1 also omits a separate skip branch for no relation, leaving edge construction after abstention to be checked in the implementation.
Label alignment, pretrained knowledge, and threshold selection remain part of the system: \(\tau=0.3\) comes from a trade-off on a GPT-4o-annotated PSG subset. Open-world results do not establish reliable mapping-free behavior for arbitrary external categories, and unseen does not prove absence from backbone pretraining. The reconciliation of PSG segmentation annotations with the box-based implementation still needs clarification.
Reporting conflicts in the original submission need to be distinguished: Section 4.4 claims GG-CRD gains of 1.54/1.92, but the last two Table 5 rows differ by 0.69/1.92 in zR@50/100. Section 4 calls downstream metrics accuracy, whereas Table 4 uses a unified score whose 3.27-point increase should not be conflated with accuracy. Appendix C.1 writes LLaVA-OneVision-1.5-7B, while Figure 4 and Appendix A use LLaVA-OneVision-1.5-8B; the discrepancy is preserved without reassigning that paragraph's average gains.

Reading Limitations: This note covers the original submission's main text and appendices, without verifying a code implementation. The per-benchmark bar values in Figure 4 have not been checked individually, so no precise gains are inferred from them. These limits describe the note's verification scope, not defects in the paper.

vs OwSGG: Both concern VLM-based open-world scene graphs. HSGG's explicit additions combine hierarchical entity expansion, geometric filtering, and relation-level contrastive decoding; system-level table differences do not isolate the superiority of any one component over its counterpart.
vs PGSG, CAPSGG, and OvSGTR: These comparisons help contextualize open-vocabulary or zero-shot SGG, but training regimes and components differ. HSGG demonstrates a competitive structured reasoning pipeline without additional SGG training.
vs VCD and ICD: VCD constructs contrast through visual perturbation and ICD through instruction perturbation. GG-CRD targets relational co-occurrence bias with image-free geometric and category information and explicitly elicited hallucinated predicates. An important next question is how to avoid excessive penalties when a correct relation overlaps with the hallucination prior.

Rating¶

These are the reader's subjective assessments of the original submission read here: 3 means adequate, 4 strong, and 5 outstanding; half points are allowed. Writing Quality refers to the original paper, and Value concerns research contributions and transfer potential, not deployment readiness.

Novelty: 5/5. Uses geometry both to filter candidates and to construct hallucination priors, linking these roles to bidirectional hierarchical entity perception in a distinctive mechanism targeting relational errors.
Experimental Thoroughness: 4.5/5. Zero-shot and open-world SGG, component ablations, VQA, and image captioning provide complementary evaluations; quantitative efficiency, silver-standard reliability, and label-alignment details need further support.
Writing Quality: 4.5/5. The algorithm, core equations, and appendix prompts correspond well enough to trace the main inference process, despite localized inconsistencies in model names and ablation gains in the original submission.
Value: 5/5. Constructing reference distributions around specific relational errors has cross-task potential, and hierarchical scene graphs support downstream reasoning; practical benefits still depend on backbone quality and inference cost.