Synthetic Visual Genome 2: Extracting Large-scale Spatio-Temporal Scene Graphs from Videos¶

Conference: ECCV2026
Paper: ECCV Paper
Project: SVG2
Area: Video Understanding
Keywords: spatio-temporal scene graphs, panoptic trajectories, open vocabulary, dual resamplers, synthetic annotations

TL;DR¶

SVG2 scales automatic annotation of real videos to 636K videos and trains the trajectory-conditioned TraSeR with segmentation alignment and global/local dual resamplers, achieving 79.0% object accuracy and 27.1% attribute recall on SVG2test, although its relation predictions do not consistently surpass GPT-5.

Background & Motivation¶

Video scene graphs must explain not only what appears, but also which entities interact and when those interactions occur. Nodes represent objects and their attributes, directed edges represent interactions, and video requires node identities to remain consistent across frames. For example, a person may first ride a bicycle and then push it: the categories remain unchanged, but the relations require different temporal intervals. Visual Genome and subsequent image scene graphs provide static structural supervision without directly resolving identity continuity and changing relations. Existing video datasets typically require expensive manual annotation; sparse frame sampling misses newly appearing objects, while fixed categories and long-tailed relations constrain open-vocabulary generalization.

Two connected bottlenecks arise: obtaining large amounts of fine-grained graph supervision and identifying which visual trajectory each linguistic description belongs to. Adding automatic descriptions alone is insufficient because upstream identity switches attach object names, attributes, and relations to the wrong entities. Feeding every frame's visual tokens into a large model is also problematic, as long videos and multiple objects introduce extensive token repetition. Conversely, compressing an entire trajectory into a single summary helps recognition but can erase when contact begins or ends. Dataset construction therefore needs reliable object discovery and verification, while the prediction model must preserve both stable identity and local temporal changes.

The authors consequently present two components: a foundation-model pipeline for dense temporal graphs and a smaller model trained to generate that structure. Here, synthetic primarily describes the annotations; the videos come from SA-V and PVD, rather than a video generation model. SVG2test additionally supplies a manually annotated open-vocabulary diagnostic set, avoiding evaluation solely within existing closed categories. This explains why the paper studies tracking and annotation quality alongside token organization and relation localization, rather than simply introducing a larger dataset. Core Idea: establish scalable structural supervision through object trajectories, then separately compress global semantics and local temporal evidence so that a language model can generate scene graphs grounded in identifiable entities and time intervals.

Method¶

Overall Architecture¶

The data pipeline first generates panoptic trajectories, then attaches object names, attributes, and temporally localized relations to construct SVG2. TraSeR takes video and existing object-mask trajectories, binds visual tokens to objects, compresses them through dual resamplers, and decodes a structured scene graph with a language model. The two processes have distinct responsibilities: the expensive multi-model annotation chain creates supervision, whereas the trained TraSeR performs trajectory-conditioned prediction. Figure 3 explicitly states that TraSeR does not jointly discover and track objects, so a single forward pass excludes the full cost of obtaining trajectories from raw video.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    Input["Source videos"] --> Synthesis["Trajectory-Driven<br/>Annotation Synthesis"]
    Synthesis -->|Training-sample mask trajectories| Tokens["Trajectory-Aligned<br/>Token Arrangement"]
    Given["Video and given trajectories"] --> Tokens
    Tokens --> Resample["Global and Local<br/>Dual Resampling"]
    Resample --> Decoder["Language-model decoding"]
    Synthesis -.->|Scene-graph training supervision| Decoder
    Decoder --> Output["Objects, attributes,<br/>and temporal relations"]

The given trajectories come from ground truth in the main scene-graph evaluation and from the automatic pipeline's first phase in the video QA experiment. Annotation synthesis is not a semantic teacher chain that TraSeR must invoke on every prediction, although deployment still requires an upstream trajectory source. The output preserves object IDs, semantic labels, and relation intervals rather than merely producing a natural-language video summary.

Key Designs¶

1. Trajectory-Driven Annotation Synthesis: establish entities before naming and relating them

The first phase uses SAM2 with multi-scale grid prompts to produce per-frame mask candidates, followed by online and offline tracking stages. The online stage propagates existing masks and checks regions not covered by current tracks; sufficiently large new candidates trigger new object IDs. This addresses SAM2's inability to proactively discover later objects using only its initial prompts. A conservative asymmetric-overlap matching strategy reduces identity switches during occlusion and reappearance. The offline stage replays the video, initializes each object at its first appearance, and jointly tracks all instances forward. Redundant overlapping tracks are then filtered and small mask artifacts corrected, providing more stable spatial support for subsequent descriptions. The SA-V subset also propagates human masks and fuses them with automatic trajectories, retaining synthetic masks in uncovered regions; the entire training set therefore cannot be described as free of human information.

In the second phase, DAM-3B-Video describes each trajectory, and GPT-4.1-nano extracts object names and adjectival attributes such as color and shape. SAM3 subsequently uses these labels as prompts to generate verification trajectories, matched to the original tracks through video-level spatiotemporal IoU and Hungarian assignment. It serves only as a label verifier: original masks are retained, and objects without supporting matched trajectories are discarded rather than replacing the mask collection with SAM3 outputs. The third phase supplies sampled frames, object IDs, names, and bounding boxes to GPT-5 for inter-object spatiotemporal relation inference. The seven relation types are spatial, functional, stateful, motion, social, attentional, and event-level, covering contact, carrying, relative motion, and extended events. Because spatial relations can dominate, the authors query spatial and non-spatial relations separately and suppress simple two-dimensional relations such as left of and right of that follow directly from mask layout. This directs annotation effort toward visual semantics, but does not eliminate teacher errors in crowded scenes, occlusion, or event interpretation.

2. Trajectory-Aligned Token Arrangement: assign visual evidence using mask coverage

A Qwen2.5-VL visual token corresponds to a spatially and temporally merged pixel region, not an inherently identified object. The method average-pools each object's mask over the exact spatial pixel support of a token to obtain per-frame coverage. It then takes the maximum across the consecutive frames covered by that token, avoiding dilution when the object appears in only some of those frames. Only tokens whose coverage reaches \(\tau_{\mathrm{eff}}=0.5\) are assigned to the corresponding object trajectory. The spatial pooling kernel and stride must match the visual token's actual pixel footprint, otherwise mask coordinates and model representations become misaligned. This is more direct than telling the language model where a bounding box lies, because the object's visual content is organized into its input stream first.

Assigned tokens are sorted temporally within each object, and special [TRJ] tokens separate different trajectories. The model consequently receives a sequence with explicit entity boundaries instead of having to infer which object each reference denotes from an undifferentiated video stream. Coverage selection is performed independently for each object; the paper does not require a mutually exclusive assignment of all tokens across objects. Fine-grained parts and larger objects can share visual regions, making subsequent compression important for controlling repetition and sequence length. Equations (1) through (5) are corrupted in the available text extraction; this note explains the operations from adjacent prose rather than presenting guessed repairs as the authors' exact equations.

3. Global and Local Dual Resampling: use different temporal scales for stable categories and dynamic relations

Passing every arranged token directly to the language model lets long tracks, densely sampled frames, and repeated regions consume excessive attention and computation. The object-trajectory resampler uses learnable queries in a Perceiver-Resampler to summarize an object's visual evidence across its entire temporal extent. Trajectories of different lengths become fixed-size global summaries, useful for recognizing the object and its temporally stable visual semantics. The cost is reduced temporal resolution: recognizing a bicycle does not establish when a person starts riding it. The paper therefore does not use global aggregation as its only visual representation.

The temporal-window resampler divides the video into non-overlapping windows of length \(\Delta t\) and separately summarizes each object's tokens within each window. No window summary is produced when an object is absent, making representation length adaptive to actual presence. Timestamp embeddings are inserted before each temporal resampled token so the language model can associate local changes with time intervals. Each object's final representation combines its global summary with its temporally ordered local summaries. Both branches read that object's aligned visual tokens; the local branch does not merely compress a global summary that has already lost temporal information. This distinction matters for the ablation: using only the global branch damages relation localization, whereas temporal windows can distinguish different phases of the same object's trajectory. The main text does not specify the numerical window length, so these descriptions alone do not reproduce every temporal compression setting.

A Worked Example¶

Consider a person who first rides a bicycle and then dismounts to push it; this illustrates the mechanism and is not an additional quantitative experiment from the paper. The automatic pipeline establishes tracks for the person and bicycle, assigning the bicycle a new ID through online discovery if it enters later. Object description should preserve both identities, while relation extraction must assign riding and pushing to different time intervals. At inference time, TraSeR selects visual tokens from these existing trajectories and organizes them by trajectory boundaries. Global summaries retain the person and bicycle categories, while local windows preserve evidence of the transition from riding posture to pushing. The language model can then output relations with entity endpoints and temporal ranges instead of describing riding as true throughout the video. If the upstream tracker confuses the person with another individual, the semantic model has not been shown to automatically repair that identity error.

Loss & Training¶

TraSeR is built on Qwen2.5-VL-3B and trained on a mixture of SVG2 and manually annotated academic datasets. LV-VIS, OVIS, and VIPSeg contribute video instance information, while VidOR and VidVRD supply relation supervision. Bounding-box annotations are converted to segmentation masks using SAM2, and task templates serialize heterogeneous annotations into a unified autoregressive output format. The main text describes this training format but does not introduce a separate new loss equation requiring reconstruction. Each of the two Perceiver-Resamplers has three layers and 32 learnable queries. Training freezes Qwen2.5-ViT while updating the vision-language projector, both resamplers, and the language model. Their respective learning rates are \(5\times10^{-5}\), \(1\times10^{-4}\), and \(2\times10^{-5}\), allowing the new modules to adapt more rapidly to trajectory compression. The main model is trained for one epoch, totaling 37K steps, on a single node with 8 H100 GPUs. Architectural ablations instead train for three epochs using only the SVG2 SA-V subset, so their scores must not be conflated with the main-model setting.

Key Experimental Results¶

Main Results¶

Table 2 on page 11 uses given ground-truth trajectories and a temporal IoU threshold of 0.5 for relations and triplets; values are percentages. The following selection reports all four SVG2test metrics; baselines receive bounding-box information from the trajectories, whereas TraSeR uses segmentation-mask alignment. The comparison therefore controls trajectory provenance, but does not give every model an identical representation of trajectory information.

Model	Triplet Recall	Relation Recall	Object Accuracy	Attribute Recall
GPT-4.1	6.4	7.5	58.5	15.8
GPT-5	17.9	19.4	65.5	24.1
Qwen2.5-VL-3B	0.2	0.3	24.2	1.4
FT-Qwen2.5-VL-3B (full bbox trajectories)	1.4	3.0	46.1	13.4
TraSeR	16.7	18.7	79.0	27.1

Object accuracy evaluates trajectory labels; attribute and relation recall measure matched ground-truth elements, while triplets additionally require correct subjects, objects, and predicates together. The table uses lenient semantic matching: GPT-4o-mini serves only as a pairwise lexical matcher, distinguishing identical, synonymous, hypernym/hyponym, semantically overlapping, and mismatched terms. Every category except mismatch counts as semantically correct, and relations must additionally satisfy temporal IoU requirements; this is looser than exact string matching and is not strict category recognition performance. Table 3 checks agreement with humans on 250 objects and 150 relations, obtaining lenient Cohen's \(\kappa\) values of 0.877 and 0.791, respectively. This supports the evaluator's usefulness but does not constitute human verification of all long-tailed categories or temporal localization.

Ablation Study¶

Table 4 on page 12 trains for three epochs on the SA-V subset; the following selection uses only its PVSG* columns, with all metrics in percentages. The original caption notes that variants without resamplers cannot support long videos, so PVSG uses its VidOR subset rather than the complete PVSG evaluation in Table 2.

Temporal-Window Resampler	Object-Trajectory Resampler	Alignment Signal	Triplet Recall	Relation Recall	Object Accuracy
No	No	Segmentation	7.03	7.15	73.34
No	Yes	Segmentation	2.44	2.68	77.10
Yes	No	Segmentation	8.52	8.63	77.84
Yes	Yes	Bounding boxes	8.69	9.39	73.05
Yes	Yes	Segmentation	10.14	11.38	78.21

Global object aggregation alone achieves 2.68 relation recall; restoring temporal windows raises it to 11.38, showing that compression must preserve local temporal structure. With both resamplers present, segmentation alignment improves object accuracy from 73.05 to 78.21 compared with bounding-box alignment. However, Table 4 reports VidOR relation recall of 16.78 for temporal windows alone versus 16.30 for both branches, so the best combined configuration is not best in every column.

Table 6 on page 14 evaluates GPT-4.1 using video frames at 1 fps, sampling 1K open-ended AGQA 2.0 questions and 500 Perception-Test multiple-choice questions. Trajectories are automatically generated before scene-graph prediction, making this closer to a complete application pipeline than ground-truth-trajectory evaluation; the metric is percentage accuracy.

Input	AGQA 2.0	Perception-Test
Video only	25.9	66.8
Video + Qwen2.5-VL-3B scene graphs	24.8	68.6
Video + TraSeR scene graphs	26.3	71.4

Key Findings¶

On SVG2test, TraSeR exceeds GPT-5 by 13.5 percentage points in object accuracy and 3.0 points in attribute recall, but remains slightly lower in triplet and relation recall.
Video QA gains over video alone are 0.4 and 4.6 percentage points; gains over Qwen scene-graph inputs are 1.5 and 2.8 points, so the comparison baseline must be explicit.
Manual annotation verification on page 6 samples 1.2K object trajectories, 2K attributes, and 1.1K relations, yielding accuracies of 93.8%, 88.3%, and 85.4%; an additional 350 complete triplets yield 84.3% accuracy.
The full training configuration in Table 5 reports PVSG triplet/relation scores of 16.91/16.13, whereas Table 2 reports 16.1/16.9; the paper does not explain this discrepancy, and this note does not force the tables into agreement.

Highlights & Insights¶

Segmentation evidence establishes object identity before input reaches the language model. This is a more direct structural constraint than extending prompts and relying on coordinate text to maintain cross-frame references.
The two resamplers serve stable and dynamic semantics separately. Compression judged only by token count can miss temporal information loss, which the relation-localization ablation exposes.
The SAM3 verifier checks visual support for labels without replacing original masks. This separates geometric trajectory consistency from confidence in object naming.
More graph content is not automatically better: Qwen graphs reduce AGQA from 25.9 to 24.8. A reasonable reader interpretation is that intermediate-representation accuracy matters more than adopting a graph format itself.

Limitations & Future Work¶

The authors explicitly acknowledge dependence on upstream trajectories and leave unified object proposal, tracking, and scene-graph generation to future work; current scores do not measure end-to-end detection and tracking performance.
Annotations inherit foundation-model errors: about 86% of manually inspected relation errors involve incorrect predicates under occlusion or crowding, and 14% involve temporal ranges; scaling alone does not remove these biases.
SVG2test contains only 100 videos, and lenient semantic evaluation accepts hierarchy and partial overlap; conclusions should remain tied to this diagnostic set and protocol rather than arbitrary open-world accuracy.
Multiple appendix references appear as Appx. ??, the cache contains no corresponding appendix, and equations are damaged in extraction; extended strict-match results and settings such as window length cannot be independently checked here.
Full data-synthesis costs and end-to-end inference latency are not reported, so a small model's single forward pass does not establish that the complete application pipeline is inexpensive.
The authors also identify dual-use risks in surveillance and large-scale behavioral analysis; deployment should constrain video provenance, identifiable personal information, and uses of structured relational data.

Visual Genome / Synthetic Visual Genome: the former establishes image-level object, attribute, and relation resources, while the latter scales automatic image annotation; SVG2 adds object persistence, appearance/disappearance, and relation intervals.
PVSG / VidOR: these provide important manually annotated video-relation benchmarks but have limited category and attribute coverage; SVG2 complements them with open vocabulary, dense trajectories, and attribute supervision rather than simply replacing human ground truth.
SAM2 / DAM / SAM3: these contribute trajectories, localized descriptions, and label verification, respectively. The contribution lies largely in combining them into stable temporal annotation and using the results to train a dedicated predictor.
Flamingo's Perceiver-Resampler: this provides the underlying learnable-query compression mechanism; TraSeR adapts it to entity trajectories and temporal windows rather than introducing an entirely new attention operator.
Research direction: adaptive windows around relation boundaries could prioritize interaction changes under a limited token budget; this is a reader proposal, not a result evaluated by the paper.

Rating¶

Novelty: 4/5. Large-scale trajectory annotation and object/temporal representations are coherently combined, while underlying visual and resampling mechanisms largely reuse existing models.
Experimental Thoroughness: 4/5. Structural prediction, manual verification, architecture/data ablations, and downstream QA are covered, but diagnostic scale and cost reporting remain limited.
Writing Quality: 3/5. Task boundaries and method logic are clear, but missing appendix references, extracted equations, and cross-table discrepancies complicate verification.
Value: 4/5. The work offers useful open-vocabulary video structural supervision, provided trajectory quality, teacher biases, and privacy boundaries remain explicit.