Skip to content

Inter-3D VQA: A Roadside Multimodal Benchmark for 3D Spatiotemporally Grounded Visual Question Answering

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/ASU-Suo-Lab/Inter-3D-VQA
Area: Multimodal VLM
Keywords: Roadside Perception, 3D Spatiotemporal Grounding, Visual Question Answering, Multimodal LLM, Intersection Understanding

TL;DR

Inter-3D VQA establishes the first large-scale roadside multimodal benchmark for 3D spatiotemporally grounded VQA (407K QA pairs with dual referring formats), coupled with Inter-Geo, an MLLM leveraging dual-branch LiDAR querying (DQNet) and gated decoder-level adaptation (LDA) to dramatically outperform existing VLMs in real-world metric reasoning and trajectory forecasting.

Background & Motivation

Recent advances in multimodal large language models (MLLMs) have revolutionized traffic scene understanding, transitioning intelligent transportation systems from isolated perception tasks toward open-ended, language-driven reasoning. Existing visual question answering (VQA) benchmarks in transportation are heavily biased toward ego-centric viewpoints mounted on autonomous vehicles, such as NuScenes-QA, DriveLM, and NuPlanQA. However, ego-vehicle sensors inevitably suffer from severe local occlusion, blind spots, and limited observation horizons, making it difficult to capture comprehensive spatial interactions and macro-level traffic dynamics at complex, conflict-prone intersections.

To overcome the inherent constraints of onboard sensing, roadside infrastructure systems provide a panoramic vantage point for holistic traffic monitoring and safety management, fostering roadside datasets like SUTD-TrafficQA, TUMTraffic-VideoQA, and RoadSceneVQA. Nonetheless, these existing roadside benchmarks are fundamentally constrained to single-view 2D images or 2D video streams. They lack explicit 3D geometric depth and high-definition infrastructure topology. Consequently, when evaluated on critical downstream queries—such as computing the exact metric distance of a pedestrian to a crosswalk exit, predicting future trajectories of turning vehicles, or assessing near-miss collision risks—models are forced to guess from 2D pixel coordinates, frequently suffering from severe hallucinations and ungrounded approximations.

Bridging this gap requires a standardized benchmark that unifies synchronized multi-view imagery, dense LiDAR point clouds, and high-definition road topology, supported by rigorous multimodal evaluation protocols. A viable benchmark must not only supply metric-accurate 3D physical ground truth across camera views, but also replace conventional surface lexical metrics with holistic evaluation encompassing textual consistency, numerical accuracy, and structural semantic correctness. Core idea: this work introduces Inter-3D VQA, the first large-scale roadside multimodal benchmark pairing synchronized multi-view images with global LiDAR point clouds across 407K QA pairs, alongside Inter-Geo, an MLLM that injects dual-branch object and scene geometric priors via gated decoder adapters, and Inter-Metrics, a tri-fold evaluation protocol.

Method

Overall Architecture

The Inter-3D VQA pipeline and the Inter-Geo model are designed to realize geometry-faithful 3D multimodal understanding across complex four-way intersections. The dataset construction pipeline synchronizes four directional camera views (North, South, East, West) with two 128-beam LiDAR point clouds and HD lane maps sampled at 2 Hz, extracting structured environmental, object-level, and infrastructure metadata. Through expert-curated template expansion and domain LLM diversification, the dataset instantiates both free-form and explicit grounded questions, filtered via multi-stage sanity checks. On the model side, Inter-Geo preserves pretrained vision-language representations in Qwen3-VL-4B while using an off-the-shelf 3D backbone (LION with TransFusion head) to extract instance-level and scene-level BEV features, which are compressed via Dual Querying Net (DQNet) and injected into the language decoder via LiDAR Decoder Adapters (LDA).

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    In["Multimodal Inputs<br/>Four Camera Views + Global LiDAR + Topology"] --> Stage1["Multi-source Metadata Curation & Dual QA Instantiation<br/>Free-form Topology vs Grounded 3D-Aware Tuples"]
    Stage1 --> Stage2["Multi-stage Sanity Check & Quality Filtering<br/>Traffic Commonsense Filter & Layered Balancing"]
    Stage2 --> Stage3["Dual Querying Net DQNet<br/>DETR Object Queries & Perceiver Scene Bottleneck"]
    Stage3 --> Stage4["LiDAR Decoder Adapter LDA<br/>Cross-attention Retrieval & Gated Late Residual Injection"]
    Stage4 --> Stage5["Inter-Metrics Evaluation Protocol<br/>Text Consistency + Metric Numerical Error + Semantic Exact Match"]

Key Designs

1. Multi-source Metadata Curation & Dual QA Instantiation: Resolving Ambiguity via Infrastructure Topology and 3D Tuples

To eliminate referring ambiguities inherent to 2D roadside cameras while providing rigorous spatial anchors, the benchmark formulates two complementary question formats grounded in synchronized multi-view feeds, dual 128-beam LiDAR, and HD maps: - Free-form QA: Embeds infrastructure topology directly into natural language descriptions, identifying participants via structural constraints such as "the second bus in the through lane on the west approach" or "the pedestrian on the north crosswalk," resolving object ambiguity purely through linguistic topological context. - Grounded QA: Employs an explicit 3D-aware spatiotemporal reference tuple \(\langle o, fn, x_{3d}, y_{3d}, \text{camview}, x_{2d}, y_{2d}, \dots \rangle\), where \(o\) is a scene-stable object ID, \(fn\) denotes the temporal offset, \((x_{3d}, y_{3d})\) are ground-plane metric coordinates, and each \((\text{camview}, x_{2d}, y_{2d})\) tuple maps the 2D bounding box center in each visible camera view, establishing a mathematically rigorous bridge across modalities.

2. Multi-stage Sanity Check & Quality Filtering: Eliminating Commonsense Hallucinations and Long-Tail Imbalance

Unchecked automated template instantiation frequently generates invalid physical scenarios or rule-violating queries. The construction pipeline applies a four-stage quality assurance protocol: - Commonsense and Traffic Rule Validation: Strips out illogical combinations (e.g., vehicles executing unauthorized maneuvers or pedestrians on inaccessible highway ramps); - Disambiguation Filtering: Removes questions yielding multiple legitimate answers or ambiguous semantic referents; - LLM Semantic Diversification: Deploys domain-adapted LLMs to synthesize linguistically diverse paraphrases while strictly preserving slot semantics; - Sample Auditing & Balancing: Enforces balanced sampling across categories and instances. Crucially, near-miss events, queuing dynamics, and multi-agent conflicts comprise 16.8% of the final 407K dataset, with over 30% covering active speeding or aggressive lane shifts.

3. Dual Querying Net (DQNet): Decoupled Object- and Scene-Level Geometric Compression

Directly flattening raw 3D LiDAR tokens into an LLM context window causes quadratic attention costs and overwhelms language representations. DQNet processes variable-length raw 3D encoder outputs (\(X_o \in \mathbb{R}^{N_o \times 523}\) and \(X_s \in \mathbb{R}^{H_b \times W_b \times 386}\)) through two distinct branches: - Object Branch: Adopts DETR-style queries \(Q_a\) that attend to projected object tokens \(T_o\), followed by a learnable confidence gate \(W_c\) to suppress detection noise: $\(Z_o = \sigma(W_c \hat{Q}_a) \odot \hat{Q}_a, \quad \text{where } \hat{Q}_a = \text{MLP}(\text{Attn}_{\text{self}}(\text{Attn}_{\text{cross}}(Q_a, T_o)))\)$ retaining the top 16 most confident instance representations. - Scene Branch: Utilizes a Perceiver-style latent bottleneck where 64 learnable queries \(Q_s\) attend to the projected bird's-eye-view feature map \(T_s\): $\(Z_s = \text{MLP}(\text{Attn}_{\text{cross}}(Q_s, T_s))\)$ condensing dense lane layouts and intersection geometry into 64 highly expressive tokens.

4. LiDAR Decoder Adapter (LDA): Non-Disruptive Late-Stage Gated Injection

Rather than prepending LiDAR tokens at the input level or fusing them early into visual embeddings—which damages the pretrained vision-language alignment manifold—Inter-Geo inserts LDAs into the final 4 layers of the Qwen3-VL decoder. Given decoder hidden states \(H\), LDA computes cross-attention against the frozen geometric tokens \(\Delta_{s/o} = \text{Attn}_{\text{cross}}(H, Z_{s/o})\) and scales them with input-conditioned gates: $\(G_{s/o} = \sigma(\text{MLP}_{s/o}(H))\)$ The residual update is formulated as: $\(H_{\text{out}} = H + \text{MLP}_f([G_s \odot \Delta_s; G_o \odot \Delta_o])\)$ Gate biases are initialized to \(-5.0\), ensuring that unaligned 3D signals do not destabilize the pretrained language model during initial training steps.

5. Inter-Metrics Evaluation Protocol: Tri-Fold Evaluation of Text, Physical Metrics, and Traffic Semantics

To prevent models from exploiting superficial lexical overlap while failing spatial physics, Inter-Metrics unifies three evaluation axes: - Textual Consistency (\(S_{\text{text}}\)): A weighted combination of Normalized Exact Match, BLEU-4, ROUGE-L F1, BERTScore, and SimCSE sentence embeddings: $\(S_{\text{text}}(\hat{y}, y) = \sum_{m \in M} w_m s_m(\hat{y}, y)\)$ - Numerical Accuracy (\(S_{\text{num}}\)): Evaluates scalars via absolute error \(|e_i|\) and vectors (3D coordinates, waypoints) via Euclidean distance \(\|e_i\|_2\), normalized by domain thresholds \(\tau_i\): $\(S_{\text{num}} = 1 - \frac{1}{N} \sum_{i=1}^N \min\left(\frac{e_i}{\tau_i}, 1\right)\)$ where thresholds are calibrated to count (\(\tau=2\)), distance (\(\tau=10\,\text{m}\)), speed (\(\tau=5\,\text{m/s}\)), acceleration (\(\tau=1\,\text{m/s}^2\)), 3D coordinates (\(\tau=8\,\text{m}\)), and waypoint offsets (\(\tau=5\,\text{m}\)). - Semantic Correctness (\(S_{\text{sem}}\)): Performs exact matching over structured semantic fields (lane assignment, movement direction, conflict presence): $\(S_{\text{sem}}(\hat{z}, z) = \frac{1}{|F|} \sum_{f \in F} \mathbb{I}(\hat{z}_f = z_f)\)$

Loss & Training

Inter-Geo fine-tunes Qwen3-VL-4B-Instruct using LoRA (rank 8, scaling factor 16, dropout 0.05) via Llama-Factory. The vision resolution is capped at 262,144 pixels with a maximum sequence length of 4,096. Training runs on four NVIDIA RTX PRO 6000 Blackwell GPUs in BF16 precision for 3 epochs, with a per-GPU batch size of 1 and gradient accumulation of 8 (effective batch size 32), using a cosine learning rate scheduler and an initial learning rate of \(1 \times 10^{-4}\).

Key Experimental Results

Main Results

The authors conduct comprehensive benchmarking across open-source generalist VLMs, specialized driving VLMs, roadside models, closed-source API models, and Inter-Geo under identical prompts, data splits, and evaluation criteria.

Table 1: Performance comparison across VLM families on Free-form and Grounded Inter-3D VQA

Model Type Model Name Free-form \(S_{\text{text}} \uparrow\) Free-form \(S_{\text{num}} \uparrow\) Free-form \(S_{\text{sem}} \uparrow\) Grounded \(S_{\text{text}} \uparrow\) Grounded \(S_{\text{num}} \uparrow\) Grounded \(S_{\text{sem}} \uparrow\)
Open-source Generalist LLaVA-NeXT-8B 0.788 0.347 0.646 0.803 0.341 0.656
Open-source Generalist Llama3.2-11B 0.770 0.289 0.583 0.788 0.364 0.576
Open-source Generalist Qwen3-VL-4B 0.806 0.397 0.709 0.827 0.432 0.736
Open-source Generalist Qwen3-VL-8B 0.821 0.457 0.749 0.851 0.488 0.802
Open-source Generalist InternVL3-4B 0.794 0.363 0.656 0.811 0.405 0.666
Open-source Generalist InternVL3-8B 0.814 0.435 0.724 0.841 0.484 0.768
Specialized Driving Senna-VLM 0.784 0.321 0.628 0.808 0.402 0.644
Specialized Driving NuScenes-QA 0.671 0.288 0.294 0.612 0.127 0.173
Specialized Driving OmniDrive 0.752 0.336 0.560 0.731 0.286 0.440
Specialized Driving OpenDriveVLA 0.777 0.421 0.624 0.774 0.337 0.534
Specialized Driving DriveLM 0.766 0.381 0.582 0.755 0.272 0.481
Specialized Driving BEV-LLM 0.772 0.376 0.613 0.761 0.301 0.507
Specialized Roadside TraffiX-Qwen 0.753 0.322 0.564 0.743 0.264 0.495
Ours Inter-Geo (4B) 0.820 0.482 0.748 0.859 0.589 0.828

Table 2: Breakdown of normalized numerical error across physical fields on Grounded Inter-3D VQA (Lower is better \(\downarrow\))

Model Name Count \(\downarrow\) Distance \(\downarrow\) Speed \(\downarrow\) Acceleration \(\downarrow\) Coord3d \(\downarrow\) Coord2d \(\downarrow\) Waypoint \(\downarrow\)
LLaVA-NeXT-8B 0.35 0.24 0.69 0.94 0.81 0.87 0.71
Qwen3-VL-4B 0.26 0.18 0.53 0.64 0.77 0.81 0.78
Qwen3-VL-8B 0.22 0.15 0.37 0.73 0.73 0.73 0.66
InternVL3-8B 0.25 0.18 0.38 0.54 0.77 0.78 0.71
Senna-VLM 0.28 0.24 0.61 0.63 0.81 0.87 0.74
TraffiX-Qwen 0.37 0.69 0.49 1.00 0.91 0.89 0.80
Inter-Geo (4B) 0.22 0.15 0.29 0.59 0.48 0.76 0.31

Ablation Study

Table 3: Ablation of LiDAR-related architectural components on Grounded Inter-3D VQA

3D Encoder DQNet Object Branch DQNet Scene Branch LDA Adapter \(S_{\text{text}} \uparrow\) \(S_{\text{num}} \uparrow\) \(S_{\text{sem}} \uparrow\)
Full (LION) \(\checkmark\) \(\checkmark\) \(\checkmark\) 0.859 0.589 0.828
Replace with MLP \(\checkmark\) \(\checkmark\) \(\checkmark\) 0.841 0.477 (-19.0%) 0.765 (-7.6%)
Full (LION) \(\times\) \(\checkmark\) \(\checkmark\) 0.857 0.566 (-4.0%) 0.822 (-0.7%)
Full (LION) \(\checkmark\) \(\times\) \(\checkmark\) 0.857 0.553 (-9.5%) 0.817 (-1.3%)
Full (LION) \(\checkmark\) \(\checkmark\) \(\times\) 0.827 0.432 (-26.7%) 0.736 (-11.1%)

Table 4: Ablation on modality injection strategy and 3D perception backbones

Experiment Category Configuration Injection Stage / Backbone Type \(S_{\text{text}} \uparrow\) \(S_{\text{num}} \uparrow\) \(S_{\text{sem}} \uparrow\)
Injection Strategy Input Embeddings Input sequence concatenation 0.856 0.552 0.816
Vision Fusion Image feature cross-attention 0.850 0.549 0.810
LDA (Ours) Decoder top-4 layers gated injection 0.859 0.589 0.828
3D Encoder CenterPoint Sparse Convolution (Voxel) 0.862 0.540 0.828
TransFusion-L Sparse Convolution + Transformer 0.857 0.580 0.819
DSVT Dynamic Sparse Voxel Transformer 0.855 0.544 0.812
LION (Ours) Linear Group RNN (Mamba) 0.859 0.589 0.828

Key Findings

  • Crucial Role of 3D Geometry in Grounded Reasoning: On grounded QA, Inter-Geo (4B) achieves \(S_{\text{num}} = 0.589\), outperforming the much larger Qwen3-VL-8B (0.488) by 20.7% and the specialized Senna-VLM (0.402) by 46.5%. Most notably, trajectory waypoint prediction error drops from 0.66~0.74 to 0.31, and 3D coordinate error decreases from 0.73 to 0.48.
  • Late Decoder-level Injection Prevents Representation Collapse: Completely removing LDA causes a catastrophic 26.7% plunge in numerical accuracy. Prepending LiDAR tokens at the input layer (0.552) or early fusion in the visual encoder (0.549) significantly lags behind LDA (0.589), proving that selective, task-conditioned cross-attention in late decoder layers preserves pretrained visual-language features.
  • Scene-level BEV Context Dominates Macro Spatial Tasks: Disabling the DQNet scene branch incurs a 9.5% drop in \(S_{\text{num}}\), whereas removing the object branch drops \(S_{\text{num}}\) by only 4.0%. This reveals that intersection layout and lane topology are primary drivers of physical spatial reasoning.
  • Vision and Geometry are Indispensable: Text-only LLMs (Qwen3-8B) exhibit performance drops of 30.2% on \(S_{\text{num}}\) and 30.8% on \(S_{\text{sem}}\), demonstrating that language priors alone fail to solve grounded spatial traffic tasks. Furthermore, 2D-only roadside models like TraffiX-Qwen exhibit severe distance errors (0.69 vs 0.15 for Inter-Geo).

Highlights & Insights

  • Dual Referring Mechanism with Infrastructure Topology: Seamlessly embedding lane-level HD map semantics into free-form queries alongside a rigorous 3D tuple representation solves the persistent issue of object referring ambiguity in dense multi-camera intersections.
  • Dual Querying Bottleneck (DQNet): Compressing massive point clouds into 16 instance tokens and 64 BEV tokens provides an efficient, plug-and-play representation that avoids context flooding while preserving fine-grained 3D coordinates.
  • Negative-Bias Gated Decoder Adaptation (LDA): Initializing gate biases to \(-5.0\) in late decoder layers protects pretrained weights from being corrupted by raw 3D feature gradients during early fine-tuning stages, enabling smooth and stable convergence.

Limitations & Future Work

  • Single-Intersection Dataset Horizon: While the dataset spans extensive weather conditions, lighting variations, and traffic patterns, all data is captured at a single four-way intersection. Cross-intersection generalization to roundabouts, highway merges, and un-signalized junctions remains an open challenge.
  • Single-Frame Temporal Modeling Ceiling: Inter-Geo currently operates on single-frame synchronized multimodal pairs. Inferring fine-grained continuous physical dynamics such as vehicle acceleration remains noisy and difficult (error stays at 0.59), calling for multi-frame temporal architectures.
  • Strict Format Dependency for Metrics: The Inter-Metrics evaluation protocol relies on robust parsing of structured model responses. Unformatted or verbose outputs can trigger parsing failures that degrade semantic match scores.
  • vs NuScenes-QA / DriveLM / DriveLMM-o1: Prior autonomous driving VQA benchmarks are tied to vehicle-mounted sensors, suffering from severe blind-spot occlusions and unstable ego-motion. Inter-3D VQA offers an unoccluded infrastructure bird's-eye perspective with global coordinate consistency.
  • vs TUMTraffic-VideoQA / RoadSceneVQA: Existing roadside datasets are limited to 2D image/video frames and evaluate models via surface language metrics. Inter-3D VQA integrates synchronized 128-beam LiDAR point clouds and introduces physical metric evaluations for distances, coordinates, and trajectories.
  • vs LiDAR-LLM / BEV-LLM: Previous point cloud LLMs focus on ego-vehicle local coordinate systems. Inter-Geo proposes decoupled instance-plus-scene querying and non-intrusive late decoder adaptation, establishing a scalable blueprint for integrating 3D geometry into generalist vision-language backbones.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ [Pioneering roadside 3D grounded VQA benchmark; highly innovative dual referring format and LDA late-stage injection mechanism]
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Extensive comparisons across generalist, driving, roadside, and closed-source models with exhaustive ablations on layers, queries, and encoders]
  • Writing Quality: ⭐⭐⭐⭐⭐ [Clear structural narrative, rigorous problem formulation, well-designed mathematical equations, and rich qualitative visualizations]
  • Value: ⭐⭐⭐⭐⭐ [Crucial stepping stone for cooperative vehicle-infrastructure systems (V2X) and 3D embodied intelligence at smart intersections]