Inter-3D VQA: A Roadside Multimodal Benchmark for 3D Spatiotemporally Grounded Visual Question Answering¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/ASU-Suo-Lab/Inter-3D-VQA
Area: Multimodal VLM
Keywords: Roadside Perception, 3D Spatiotemporal Grounding, Visual Question Answering, Multimodal LLM, Intersection Understanding
TL;DR¶
Inter-3D VQA establishes the first large-scale roadside multimodal benchmark for 3D spatiotemporally grounded VQA (407K QA pairs with dual referring formats), coupled with Inter-Geo, an MLLM leveraging dual-branch LiDAR querying (DQNet) and gated decoder-level adaptation (LDA) to dramatically outperform existing VLMs in real-world metric reasoning and trajectory forecasting.
Background & Motivation¶
Recent advances in multimodal large language models (MLLMs) have revolutionized traffic scene understanding, transitioning intelligent transportation systems from isolated perception tasks toward open-ended, language-driven reasoning. Existing visual question answering (VQA) benchmarks in transportation are heavily biased toward ego-centric viewpoints mounted on autonomous vehicles, such as NuScenes-QA, DriveLM, and NuPlanQA. However, ego-vehicle sensors inevitably suffer from severe local occlusion, blind spots, and limited observation horizons, making it difficult to capture comprehensive spatial interactions and macro-level traffic dynamics at complex, conflict-prone intersections.
To overcome the inherent constraints of onboard sensing, roadside infrastructure systems provide a panoramic vantage point for holistic traffic monitoring and safety management, fostering roadside datasets like SUTD-TrafficQA, TUMTraffic-VideoQA, and RoadSceneVQA. Nonetheless, these existing roadside benchmarks are fundamentally constrained to single-view 2D images or 2D video streams. They lack explicit 3D geometric depth and high-definition infrastructure topology. Consequently, when evaluated on critical downstream queries—such as computing the exact metric distance of a pedestrian to a crosswalk exit, predicting future trajectories of turning vehicles, or assessing near-miss collision risks—models are forced to guess from 2D pixel coordinates, frequently suffering from severe hallucinations and ungrounded approximations.
Bridging this gap requires a standardized benchmark that unifies synchronized multi-view imagery, dense LiDAR point clouds, and high-definition road topology, supported by rigorous multimodal evaluation protocols. A viable benchmark must not only supply metric-accurate 3D physical ground truth across camera views, but also replace conventional surface lexical metrics with holistic evaluation encompassing textual consistency, numerical accuracy, and structural semantic correctness. Core idea: this work introduces Inter-3D VQA, the first large-scale roadside multimodal benchmark pairing synchronized multi-view images with global LiDAR point clouds across 407K QA pairs, alongside Inter-Geo, an MLLM that injects dual-branch object and scene geometric priors via gated decoder adapters, and Inter-Metrics, a tri-fold evaluation protocol.
Method¶
Overall Architecture¶
The Inter-3D VQA pipeline and the Inter-Geo model are designed to realize geometry-faithful 3D multimodal understanding across complex four-way intersections. The dataset construction pipeline synchronizes four directional camera views (North, South, East, West) with two 128-beam LiDAR point clouds and HD lane maps sampled at 2 Hz, extracting structured environmental, object-level, and infrastructure metadata. Through expert-curated template expansion and domain LLM diversification, the dataset instantiates both free-form and explicit grounded questions, filtered via multi-stage sanity checks. On the model side, Inter-Geo preserves pretrained vision-language representations in Qwen3-VL-4B while using an off-the-shelf 3D backbone (LION with TransFusion head) to extract instance-level and scene-level BEV features, which are compressed via Dual Querying Net (DQNet) and injected into the language decoder via LiDAR Decoder Adapters (LDA).
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
In["Multimodal Inputs<br/>Four Camera Views + Global LiDAR + Topology"] --> Stage1["Multi-source Metadata Curation & Dual QA Instantiation<br/>Free-form Topology vs Grounded 3D-Aware Tuples"]
Stage1 --> Stage2["Multi-stage Sanity Check & Quality Filtering<br/>Traffic Commonsense Filter & Layered Balancing"]
Stage2 --> Stage3["Dual Querying Net DQNet<br/>DETR Object Queries & Perceiver Scene Bottleneck"]
Stage3 --> Stage4["LiDAR Decoder Adapter LDA<br/>Cross-attention Retrieval & Gated Late Residual Injection"]
Stage4 --> Stage5["Inter-Metrics Evaluation Protocol<br/>Text Consistency + Metric Numerical Error + Semantic Exact Match"]
Key Designs¶
1. Multi-source Metadata Curation & Dual QA Instantiation: Resolving Ambiguity via Infrastructure Topology and 3D Tuples
To eliminate referring ambiguities inherent to 2D roadside cameras while providing rigorous spatial anchors, the benchmark formulates two complementary question formats grounded in synchronized multi-view feeds, dual 128-beam LiDAR, and HD maps: - Free-form QA: Embeds infrastructure topology directly into natural language descriptions, identifying participants via structural constraints such as "the second bus in the through lane on the west approach" or "the pedestrian on the north crosswalk," resolving object ambiguity purely through linguistic topological context. - Grounded QA: Employs an explicit 3D-aware spatiotemporal reference tuple \(\langle o, fn, x_{3d}, y_{3d}, \text{camview}, x_{2d}, y_{2d}, \dots \rangle\), where \(o\) is a scene-stable object ID, \(fn\) denotes the temporal offset, \((x_{3d}, y_{3d})\) are ground-plane metric coordinates, and each \((\text{camview}, x_{2d}, y_{2d})\) tuple maps the 2D bounding box center in each visible camera view, establishing a mathematically rigorous bridge across modalities.
2. Multi-stage Sanity Check & Quality Filtering: Eliminating Commonsense Hallucinations and Long-Tail Imbalance
Unchecked automated template instantiation frequently generates invalid physical scenarios or rule-violating queries. The construction pipeline applies a four-stage quality assurance protocol: - Commonsense and Traffic Rule Validation: Strips out illogical combinations (e.g., vehicles executing unauthorized maneuvers or pedestrians on inaccessible highway ramps); - Disambiguation Filtering: Removes questions yielding multiple legitimate answers or ambiguous semantic referents; - LLM Semantic Diversification: Deploys domain-adapted LLMs to synthesize linguistically diverse paraphrases while strictly preserving slot semantics; - Sample Auditing & Balancing: Enforces balanced sampling across categories and instances. Crucially, near-miss events, queuing dynamics, and multi-agent conflicts comprise 16.8% of the final 407K dataset, with over 30% covering active speeding or aggressive lane shifts.
3. Dual Querying Net (DQNet): Decoupled Object- and Scene-Level Geometric Compression
Directly flattening raw 3D LiDAR tokens into an LLM context window causes quadratic attention costs and overwhelms language representations. DQNet processes variable-length raw 3D encoder outputs (\(X_o \in \mathbb{R}^{N_o \times 523}\) and \(X_s \in \mathbb{R}^{H_b \times W_b \times 386}\)) through two distinct branches: - Object Branch: Adopts DETR-style queries \(Q_a\) that attend to projected object tokens \(T_o\), followed by a learnable confidence gate \(W_c\) to suppress detection noise: $\(Z_o = \sigma(W_c \hat{Q}_a) \odot \hat{Q}_a, \quad \text{where } \hat{Q}_a = \text{MLP}(\text{Attn}_{\text{self}}(\text{Attn}_{\text{cross}}(Q_a, T_o)))\)$ retaining the top 16 most confident instance representations. - Scene Branch: Utilizes a Perceiver-style latent bottleneck where 64 learnable queries \(Q_s\) attend to the projected bird's-eye-view feature map \(T_s\): $\(Z_s = \text{MLP}(\text{Attn}_{\text{cross}}(Q_s, T_s))\)$ condensing dense lane layouts and intersection geometry into 64 highly expressive tokens.
4. LiDAR Decoder Adapter (LDA): Non-Disruptive Late-Stage Gated Injection
Rather than prepending LiDAR tokens at the input level or fusing them early into visual embeddings—which damages the pretrained vision-language alignment manifold—Inter-Geo inserts LDAs into the final 4 layers of the Qwen3-VL decoder. Given decoder hidden states \(H\), LDA computes cross-attention against the frozen geometric tokens \(\Delta_{s/o} = \text{Attn}_{\text{cross}}(H, Z_{s/o})\) and scales them with input-conditioned gates: $\(G_{s/o} = \sigma(\text{MLP}_{s/o}(H))\)$ The residual update is formulated as: $\(H_{\text{out}} = H + \text{MLP}_f([G_s \odot \Delta_s; G_o \odot \Delta_o])\)$ Gate biases are initialized to \(-5.0\), ensuring that unaligned 3D signals do not destabilize the pretrained language model during initial training steps.
5. Inter-Metrics Evaluation Protocol: Tri-Fold Evaluation of Text, Physical Metrics, and Traffic Semantics
To prevent models from exploiting superficial lexical overlap while failing spatial physics, Inter-Metrics unifies three evaluation axes: - Textual Consistency (\(S_{\text{text}}\)): A weighted combination of Normalized Exact Match, BLEU-4, ROUGE-L F1, BERTScore, and SimCSE sentence embeddings: $\(S_{\text{text}}(\hat{y}, y) = \sum_{m \in M} w_m s_m(\hat{y}, y)\)$ - Numerical Accuracy (\(S_{\text{num}}\)): Evaluates scalars via absolute error \(|e_i|\) and vectors (3D coordinates, waypoints) via Euclidean distance \(\|e_i\|_2\), normalized by domain thresholds \(\tau_i\): $\(S_{\text{num}} = 1 - \frac{1}{N} \sum_{i=1}^N \min\left(\frac{e_i}{\tau_i}, 1\right)\)$ where thresholds are calibrated to count (\(\tau=2\)), distance (\(\tau=10\,\text{m}\)), speed (\(\tau=5\,\text{m/s}\)), acceleration (\(\tau=1\,\text{m/s}^2\)), 3D coordinates (\(\tau=8\,\text{m}\)), and waypoint offsets (\(\tau=5\,\text{m}\)). - Semantic Correctness (\(S_{\text{sem}}\)): Performs exact matching over structured semantic fields (lane assignment, movement direction, conflict presence): $\(S_{\text{sem}}(\hat{z}, z) = \frac{1}{|F|} \sum_{f \in F} \mathbb{I}(\hat{z}_f = z_f)\)$
Loss & Training¶
Inter-Geo fine-tunes Qwen3-VL-4B-Instruct using LoRA (rank 8, scaling factor 16, dropout 0.05) via Llama-Factory. The vision resolution is capped at 262,144 pixels with a maximum sequence length of 4,096. Training runs on four NVIDIA RTX PRO 6000 Blackwell GPUs in BF16 precision for 3 epochs, with a per-GPU batch size of 1 and gradient accumulation of 8 (effective batch size 32), using a cosine learning rate scheduler and an initial learning rate of \(1 \times 10^{-4}\).
Key Experimental Results¶
Main Results¶
The authors conduct comprehensive benchmarking across open-source generalist VLMs, specialized driving VLMs, roadside models, closed-source API models, and Inter-Geo under identical prompts, data splits, and evaluation criteria.
Table 1: Performance comparison across VLM families on Free-form and Grounded Inter-3D VQA
| Model Type | Model Name | Free-form \(S_{\text{text}} \uparrow\) | Free-form \(S_{\text{num}} \uparrow\) | Free-form \(S_{\text{sem}} \uparrow\) | Grounded \(S_{\text{text}} \uparrow\) | Grounded \(S_{\text{num}} \uparrow\) | Grounded \(S_{\text{sem}} \uparrow\) |
|---|---|---|---|---|---|---|---|
| Open-source Generalist | LLaVA-NeXT-8B | 0.788 | 0.347 | 0.646 | 0.803 | 0.341 | 0.656 |
| Open-source Generalist | Llama3.2-11B | 0.770 | 0.289 | 0.583 | 0.788 | 0.364 | 0.576 |
| Open-source Generalist | Qwen3-VL-4B | 0.806 | 0.397 | 0.709 | 0.827 | 0.432 | 0.736 |
| Open-source Generalist | Qwen3-VL-8B | 0.821 | 0.457 | 0.749 | 0.851 | 0.488 | 0.802 |
| Open-source Generalist | InternVL3-4B | 0.794 | 0.363 | 0.656 | 0.811 | 0.405 | 0.666 |
| Open-source Generalist | InternVL3-8B | 0.814 | 0.435 | 0.724 | 0.841 | 0.484 | 0.768 |
| Specialized Driving | Senna-VLM | 0.784 | 0.321 | 0.628 | 0.808 | 0.402 | 0.644 |
| Specialized Driving | NuScenes-QA | 0.671 | 0.288 | 0.294 | 0.612 | 0.127 | 0.173 |
| Specialized Driving | OmniDrive | 0.752 | 0.336 | 0.560 | 0.731 | 0.286 | 0.440 |
| Specialized Driving | OpenDriveVLA | 0.777 | 0.421 | 0.624 | 0.774 | 0.337 | 0.534 |
| Specialized Driving | DriveLM | 0.766 | 0.381 | 0.582 | 0.755 | 0.272 | 0.481 |
| Specialized Driving | BEV-LLM | 0.772 | 0.376 | 0.613 | 0.761 | 0.301 | 0.507 |
| Specialized Roadside | TraffiX-Qwen | 0.753 | 0.322 | 0.564 | 0.743 | 0.264 | 0.495 |
| Ours | Inter-Geo (4B) | 0.820 | 0.482 | 0.748 | 0.859 | 0.589 | 0.828 |
Table 2: Breakdown of normalized numerical error across physical fields on Grounded Inter-3D VQA (Lower is better \(\downarrow\))
| Model Name | Count \(\downarrow\) | Distance \(\downarrow\) | Speed \(\downarrow\) | Acceleration \(\downarrow\) | Coord3d \(\downarrow\) | Coord2d \(\downarrow\) | Waypoint \(\downarrow\) |
|---|---|---|---|---|---|---|---|
| LLaVA-NeXT-8B | 0.35 | 0.24 | 0.69 | 0.94 | 0.81 | 0.87 | 0.71 |
| Qwen3-VL-4B | 0.26 | 0.18 | 0.53 | 0.64 | 0.77 | 0.81 | 0.78 |
| Qwen3-VL-8B | 0.22 | 0.15 | 0.37 | 0.73 | 0.73 | 0.73 | 0.66 |
| InternVL3-8B | 0.25 | 0.18 | 0.38 | 0.54 | 0.77 | 0.78 | 0.71 |
| Senna-VLM | 0.28 | 0.24 | 0.61 | 0.63 | 0.81 | 0.87 | 0.74 |
| TraffiX-Qwen | 0.37 | 0.69 | 0.49 | 1.00 | 0.91 | 0.89 | 0.80 |
| Inter-Geo (4B) | 0.22 | 0.15 | 0.29 | 0.59 | 0.48 | 0.76 | 0.31 |
Ablation Study¶
Table 3: Ablation of LiDAR-related architectural components on Grounded Inter-3D VQA
| 3D Encoder | DQNet Object Branch | DQNet Scene Branch | LDA Adapter | \(S_{\text{text}} \uparrow\) | \(S_{\text{num}} \uparrow\) | \(S_{\text{sem}} \uparrow\) |
|---|---|---|---|---|---|---|
| Full (LION) | \(\checkmark\) | \(\checkmark\) | \(\checkmark\) | 0.859 | 0.589 | 0.828 |
| Replace with MLP | \(\checkmark\) | \(\checkmark\) | \(\checkmark\) | 0.841 | 0.477 (-19.0%) | 0.765 (-7.6%) |
| Full (LION) | \(\times\) | \(\checkmark\) | \(\checkmark\) | 0.857 | 0.566 (-4.0%) | 0.822 (-0.7%) |
| Full (LION) | \(\checkmark\) | \(\times\) | \(\checkmark\) | 0.857 | 0.553 (-9.5%) | 0.817 (-1.3%) |
| Full (LION) | \(\checkmark\) | \(\checkmark\) | \(\times\) | 0.827 | 0.432 (-26.7%) | 0.736 (-11.1%) |
Table 4: Ablation on modality injection strategy and 3D perception backbones
| Experiment Category | Configuration | Injection Stage / Backbone Type | \(S_{\text{text}} \uparrow\) | \(S_{\text{num}} \uparrow\) | \(S_{\text{sem}} \uparrow\) |
|---|---|---|---|---|---|
| Injection Strategy | Input Embeddings | Input sequence concatenation | 0.856 | 0.552 | 0.816 |
| Vision Fusion | Image feature cross-attention | 0.850 | 0.549 | 0.810 | |
| LDA (Ours) | Decoder top-4 layers gated injection | 0.859 | 0.589 | 0.828 | |
| 3D Encoder | CenterPoint | Sparse Convolution (Voxel) | 0.862 | 0.540 | 0.828 |
| TransFusion-L | Sparse Convolution + Transformer | 0.857 | 0.580 | 0.819 | |
| DSVT | Dynamic Sparse Voxel Transformer | 0.855 | 0.544 | 0.812 | |
| LION (Ours) | Linear Group RNN (Mamba) | 0.859 | 0.589 | 0.828 |
Key Findings¶
- Crucial Role of 3D Geometry in Grounded Reasoning: On grounded QA, Inter-Geo (4B) achieves \(S_{\text{num}} = 0.589\), outperforming the much larger Qwen3-VL-8B (0.488) by 20.7% and the specialized Senna-VLM (0.402) by 46.5%. Most notably, trajectory waypoint prediction error drops from 0.66~0.74 to 0.31, and 3D coordinate error decreases from 0.73 to 0.48.
- Late Decoder-level Injection Prevents Representation Collapse: Completely removing LDA causes a catastrophic 26.7% plunge in numerical accuracy. Prepending LiDAR tokens at the input layer (0.552) or early fusion in the visual encoder (0.549) significantly lags behind LDA (0.589), proving that selective, task-conditioned cross-attention in late decoder layers preserves pretrained visual-language features.
- Scene-level BEV Context Dominates Macro Spatial Tasks: Disabling the DQNet scene branch incurs a 9.5% drop in \(S_{\text{num}}\), whereas removing the object branch drops \(S_{\text{num}}\) by only 4.0%. This reveals that intersection layout and lane topology are primary drivers of physical spatial reasoning.
- Vision and Geometry are Indispensable: Text-only LLMs (Qwen3-8B) exhibit performance drops of 30.2% on \(S_{\text{num}}\) and 30.8% on \(S_{\text{sem}}\), demonstrating that language priors alone fail to solve grounded spatial traffic tasks. Furthermore, 2D-only roadside models like TraffiX-Qwen exhibit severe distance errors (0.69 vs 0.15 for Inter-Geo).
Highlights & Insights¶
- Dual Referring Mechanism with Infrastructure Topology: Seamlessly embedding lane-level HD map semantics into free-form queries alongside a rigorous 3D tuple representation solves the persistent issue of object referring ambiguity in dense multi-camera intersections.
- Dual Querying Bottleneck (DQNet): Compressing massive point clouds into 16 instance tokens and 64 BEV tokens provides an efficient, plug-and-play representation that avoids context flooding while preserving fine-grained 3D coordinates.
- Negative-Bias Gated Decoder Adaptation (LDA): Initializing gate biases to \(-5.0\) in late decoder layers protects pretrained weights from being corrupted by raw 3D feature gradients during early fine-tuning stages, enabling smooth and stable convergence.
Limitations & Future Work¶
- Single-Intersection Dataset Horizon: While the dataset spans extensive weather conditions, lighting variations, and traffic patterns, all data is captured at a single four-way intersection. Cross-intersection generalization to roundabouts, highway merges, and un-signalized junctions remains an open challenge.
- Single-Frame Temporal Modeling Ceiling: Inter-Geo currently operates on single-frame synchronized multimodal pairs. Inferring fine-grained continuous physical dynamics such as vehicle acceleration remains noisy and difficult (error stays at 0.59), calling for multi-frame temporal architectures.
- Strict Format Dependency for Metrics: The Inter-Metrics evaluation protocol relies on robust parsing of structured model responses. Unformatted or verbose outputs can trigger parsing failures that degrade semantic match scores.
Related Work & Insights¶
- vs NuScenes-QA / DriveLM / DriveLMM-o1: Prior autonomous driving VQA benchmarks are tied to vehicle-mounted sensors, suffering from severe blind-spot occlusions and unstable ego-motion. Inter-3D VQA offers an unoccluded infrastructure bird's-eye perspective with global coordinate consistency.
- vs TUMTraffic-VideoQA / RoadSceneVQA: Existing roadside datasets are limited to 2D image/video frames and evaluate models via surface language metrics. Inter-3D VQA integrates synchronized 128-beam LiDAR point clouds and introduces physical metric evaluations for distances, coordinates, and trajectories.
- vs LiDAR-LLM / BEV-LLM: Previous point cloud LLMs focus on ego-vehicle local coordinate systems. Inter-Geo proposes decoupled instance-plus-scene querying and non-intrusive late decoder adaptation, establishing a scalable blueprint for integrating 3D geometry into generalist vision-language backbones.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ [Pioneering roadside 3D grounded VQA benchmark; highly innovative dual referring format and LDA late-stage injection mechanism]
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Extensive comparisons across generalist, driving, roadside, and closed-source models with exhaustive ablations on layers, queries, and encoders]
- Writing Quality: ⭐⭐⭐⭐⭐ [Clear structural narrative, rigorous problem formulation, well-designed mathematical equations, and rich qualitative visualizations]
- Value: ⭐⭐⭐⭐⭐ [Crucial stepping stone for cooperative vehicle-infrastructure systems (V2X) and 3D embodied intelligence at smart intersections]